Object data processing method and device, storage medium and electronic equipment

Abstract

The application discloses a kind of object data processing method and device, storage medium and electronic equipment, involve computer field technical field, comprising: receiving data write request;In the case where write operation is completed based on object data, the data storage address and transition layer address allocated for object data are obtained, wherein the data storage address is used to indicate the writing position of object data, and the mapping relationship between the logical unit address of object data and transition layer address is determined;In the first mapping relationship set, the first mapping relationship between the logical unit address and transition layer address is added;In the second mapping relationship set, the second mapping relationship between transition layer address and data storage address is added, so that the mapping relationship between data storage address and logical unit address is effectively dissociated based on transition layer address, and then the problem of low efficiency of existing data operation is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for processing object data, a storage medium, and an electronic device. Background Technology

[0002] Centralized storage devices provide block storage space to servers. To improve storage efficiency, existing centralized storage devices have added many features, such as garbage collection. In addition, to reduce wear and tear on flash memory and extend the lifespan of storage media, most centralized storage devices also support deduplication.

[0003] Existing centralized storage devices typically rely on the mapping relationship between logical unit addresses provided by the server and data storage addresses on the storage device when executing data operations based on server data operation instructions. Centralized storage devices, based on their storage optimization functions, identify and eliminate duplicate data blocks in storage. This results in a very complex mapping relationship between logical unit addresses and data storage addresses, leading to significantly lower efficiency in performing data operations based on this mapping relationship within centralized storage devices.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method and apparatus for processing object data, a storage medium, and an electronic device to at least solve the technical problem of significantly low efficiency in the process of related data operations.

[0006] This application provides a method for processing object data, including:

[0007] Receive a data write request, which carries the logical unit address and the object data to be written;

[0008] When a write operation is completed based on object data, the data storage address and transition layer address allocated to the object data are obtained. The data storage address is used to indicate the write location of the object data, and the transition layer address determines the mapping relationship between the logical unit address of the object data and the transition layer address.

[0009] In the first mapping relationship set, add the first mapping relationship between the logical unit address and the transition layer address;

[0010] In the second mapping relationship set, a second mapping relationship between the transition layer address and the data storage address is added.

[0011] This application also provides an object data processing apparatus, including:

[0012] The receiving unit is used to receive data write requests, wherein the data write request carries the logical unit address and the object data to be written;

[0013] The acquisition unit is used to acquire the data storage address and transition layer address allocated to the object data when the write operation is completed based on the object data. The data storage address is used to indicate the write position of the object data, and the transition layer address determines the mapping relationship between the logical unit address of the object data and the transition layer address.

[0014] The first addition unit is used to add a first mapping relationship between the logical unit address and the transition layer address in the first mapping relationship set;

[0015] The second adding unit is used to add a second mapping relationship between the transition layer address and the data storage address in the second mapping relationship set.

[0016] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described object data processing methods.

[0017] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described object data processing methods.

[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described object data processing methods.

[0019] In the technical solution of this application, an efficient data management method is achieved by receiving a data write request and, after the write operation on the object-based data is completed, obtaining the data storage address and transition layer address allocated to the object data. The data storage address indicates the write location of the object data, while the transition layer address determines the mapping relationship between the logical unit address and the transition layer address of the object data. This method effectively decouples the mapping relationship between the data storage address and the logical unit address based on the transition layer address by adding a first mapping relationship between the logical unit address and the transition layer address to a first mapping relationship set, and a second mapping relationship between the transition layer address and the data storage address to a second mapping relationship set.

[0020] This strategy of dissociating mapping relationships significantly improves the flexibility and efficiency of data operations compared to existing methods that directly map logical unit addresses to data storage addresses. By introducing a transitional layer address, the system can flexibly manage and adjust mapping relationships across different sets, thereby optimizing data access paths, reducing unnecessary data movement and copying operations, and thus solving the technical problem of significantly low efficiency in related data operations. Attached Figure Description

[0021] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a hardware structure block diagram of a server device for a method of processing object data according to an embodiment of this application;

[0023] Figure 2 This is a flowchart of an object data processing method according to an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of an object data processing method according to an embodiment of this application;

[0025] Figure 4 This is a flowchart of another object data processing method according to an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the structure of an object data processing apparatus according to an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of the structure of an electronic device for processing object data according to an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0029] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0030] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a server device for a method of processing object data according to an embodiment of this application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0032] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data processing method of the memory in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0033] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0034] It should be noted that with the rapid development of flash memory technology, existing storage devices have added many features, such as garbage collection. Furthermore, to reduce wear and tear on the flash memory media and extend its lifespan, most storage devices also support deduplication.

[0035] In these storage devices, deduplication and garbage collection functions operate internally and are transparent to the servers using them. From the server's perspective, the storage device provides storage volumes of a specified size. From the storage device's perspective, when using volume storage space, the server makes access requests by specifying the starting address of the data block to be written or read, based on a data block of a specific size.

[0036] For storage devices that support deduplication, the address requested by the server cannot be directly mapped to the address where the data is actually stored. This is because when storing data, if multiple copies of the same data block exist, it is unnecessary to repeatedly store the same data on multiple media; instead, only one copy of the data block is saved, thus saving a significant amount of storage space.

[0037] Because storage devices supporting deduplication present a many-to-one relationship between logical unit addresses and actual data storage addresses, a mapping table between logical unit addresses and data storage addresses needs to be maintained within the storage device. Furthermore, since garbage collection periodically cleans up invalid data blocks in the background to free up flash memory, changes to data storage addresses may necessitate modifications to the mappings between logical unit addresses and data storage addresses, as well as the mappings between fingerprint values ​​and data storage addresses.

[0038] Therefore, in storage devices with deduplication and garbage collection capabilities, conventional metadata management methods require multiple mapping tables, making metadata management complex and consuming significant storage space. After garbage collection moves data and changes its storage location, it involves modifying multiple records in multiple mapping tables. Furthermore, it's crucial to ensure these mapping tables are correctly modified before other related requests can proceed, which significantly impacts performance. In addition, the mapping between data storage addresses and logical unit addresses often involves a one-to-many relationship, making the data structure and value modification extremely complex in key-value software design. In other words, existing data manipulation methods suffer from low efficiency.

[0039] To address the aforementioned technical problems, this application provides an optional implementation method, such as... Figure 2 As shown, the above-described object data processing method can be applied to a storage device deployed at a storage service site, including:

[0040] S202, Receive a data write request, wherein the data write request carries the logical unit address and the object data to be written;

[0041] S204, when the write operation is completed based on the object data, obtain the data storage address and transition layer address allocated to the object data, wherein the data storage address is used to indicate the write position of the object data, and the transition layer address determines the mapping relationship between the logical unit address of the object data and the transition layer address;

[0042] S206, In the first mapping relationship set, add the first mapping relationship between the logical unit address and the transition layer address;

[0043] S208, In the second mapping relationship set, add a second mapping relationship between the transition layer address and the data storage address.

[0044] In the above embodiments of this application, the data write request can be an operation request sent by a client or server to request that object data be written to the current storage device.

[0045] It is understood that the data write request in step S202 above can carry a logical unit address and the object data to be written. The logical unit address is the logical address used in the client or server system to identify the storage device, distinguishing different storage volumes or partitions. The object data can be a data entity that the client or server requests to write to the storage system; it can be a file, a portion of a file, or other types of data blocks.

[0046] Furthermore, in step S204 above, after the object data has been written to the current storage device, the data storage address and transition layer address corresponding to the object data can be obtained. The data storage address indicates the actual location of the written object data on the current storage device; the transition layer address is a configured intermediate address used to establish a mapping relationship between the logical unit address and the data storage address. Furthermore, the transition layer address can also be used to manage data storage and retrieval, as well as deduplication or garbage collection functions.

[0047] It is understood that, in the embodiments of this application, based on the aforementioned logical unit addresses, transition layer addresses, and data storage addresses, relationships can be established through a first mapping relationship set and a second mapping relationship set. The first mapping relationship set includes mapping relationships between multiple logical unit addresses and multiple transition layer addresses; these mapping relationships can be used to convert higher-level logical addresses into intermediate transition layer addresses. The second mapping relationship set includes mapping relationships between multiple transition layer addresses and multiple data storage addresses; these mapping relationships can be used to convert transition layer addresses into actual data storage addresses.

[0048] It should be noted that the above mapping relationship can be managed in the form of key-value pairs. The specific data structure management form can be one or more of the following: skip list, hash table, B+ tree, etc.

[0049] After completing the above data writing operations, corresponding mapping relationships can be added to the first and second mapping relationship sets respectively based on the current object data writing status, thereby managing the relevant address information involved in the current write operation. Furthermore, by using the transition layer address, the complex mapping relationship between logical unit addresses and data storage addresses is decoupled, improving the management efficiency of mapping relationships and thus improving the operational efficiency of performing related data operations based on mapping relationships.

[0050] It should be noted that, in the process of providing deduplication functionality, conventional storage devices need to read the existing data from the storage medium and compare each data block with the data block to be written in this request. As the scale of data storage in the storage device increases, this comparison and search method becomes very time-consuming and has significant technical problems of inefficiency.

[0051] To address the aforementioned technical problems, the storage device in this application embodiment can also solve these problems based on data fingerprint information and the mapping relationship related to the data fingerprint information.

[0052] In an optional implementation, after receiving the data write request, step S202 further includes:

[0053] S1, determine the data fingerprint information that matches the object data;

[0054] S2, after the object data has been written, add a third mapping relationship between the data fingerprint information and the transition layer address to the third mapping relationship set.

[0055] In the above embodiments, data fingerprint information can be a representation of data characteristics, typically generated using a hash function, and used to uniquely identify data content. It is understood that, in the embodiments of this application, data fingerprint information can be used to quickly compare whether data is identical, thereby improving the efficiency of deduplication operations.

[0056] The aforementioned third mapping relationship set can be used to store the mapping relationship between multiple data fingerprint information and multiple transition layer addresses. In this embodiment, when the object data has been written, the aforementioned third mapping relationship can be added to the aforementioned third mapping relationship set for subsequent data management and data retrieval operations.

[0057] Through the above-described embodiments of this application, when object data is successfully written to the storage system, the system adds a new third mapping relationship to the third mapping relationship set. This third mapping relationship associates data fingerprint information with the transition layer address, thereby establishing an index in the storage system. This allows the corresponding transition layer address to be quickly found through the data fingerprint, and thus the actual stored data to be accessed.

[0058] The data writing method based on the above implementation method will be further explained below.

[0059] In an optional implementation, before obtaining the data storage address and transition layer address allocated to the object data when the write operation is completed based on the object data, the method further includes:

[0060] S1, determine multiple sub-object data based on object data, wherein the data volume of the sub-object data is a preset size;

[0061] S2, determine the data fingerprint value corresponding to each of the multiple sub-object data, wherein the data fingerprint information matching the object data includes the data fingerprint value corresponding to each of the multiple sub-object data;

[0062] S3, write the data fingerprint value corresponding to each of the sub-object data into multiple consecutive sub-units in the first storage unit;

[0063] S4, write multiple sub-object data into multiple consecutive storage units associated with the first storage unit.

[0064] In the embodiments of this application, the minimum operation granularity of the host request can be preset, and the storage device will follow this setting when calculating the fingerprint value. Specifically, each data unit of the minimum operation granularity will generate a corresponding fingerprint value. These fingerprint values ​​will be combined to determine the overall fingerprint information of the data to be written. This granularity will also be used for comparison when performing deduplication, i.e., when determining whether the contents of two data blocks are the same. Common minimum operation granularities include 8KB, 16KB, 32KB, etc.

[0065] During data writing, these fingerprint values ​​can be saved along with the object data. In a preferred embodiment, to achieve data block alignment, a minimum disk width can be defined at the underlying storage device, with space reserved within this width for storing fingerprint values, while the remaining portion of the width is used to store multiple data at the smallest operational granularity.

[0066] In another alternative implementation, if the current storage device is configured with caching, when there are few I / O requests and the minimum data width to be written to disk is not reached, the request can wait in the cache for a period of time, such as 0.1 milliseconds. If the minimum data width to be written to disk is still not reached within this time, it can be padded with zeros to ensure that the underlying data saving operation is completed.

[0067] In the above embodiments, a metadata management method is provided, which designs a three-layer address management scheme to avoid the one-to-many management structure of the correspondence between data storage address and logical unit address, and reduces the types of relational mapping tables. The operation steps when the garbage collection function changes the data storage address are also made simpler, thereby improving the processing performance of the storage device.

[0068] The following combination Figure 3 The above data writing operations and the above mapping relationship set are explained in detail. Figure 3 This is a schematic diagram of metadata management within a current storage device.

[0069] like Figure 3 The diagram illustrates the mapping relationship between logical unit addresses, transition layer addresses, and data storage addresses.

[0070] in, Figure 3The "Logical Unit Address 1," "Logical Unit Address 2," and "Logical Unit Address 3" shown represent logical addresses in the storage system, which are used by the server when sending data write requests. The transition layer addresses X, Y, and Z are intermediate addresses between the logical unit addresses and the data storage addresses, used for managing and locating data within the storage system. Data storage addresses X2 and Y2 are the actual locations of the data on the physical storage medium. These addresses indicate the specific location where the data is written.

[0071] Furthermore, Figure 3 The "First Mapping Set" on the left describes the correspondence between logical unit addresses and transition layer addresses. Each logical unit address is mapped to a specific transition layer address (e.g., logical unit address 1 is mapped to transition layer address X, logical unit address 2 is mapped to transition layer address Y, and logical unit address 3 is mapped to transition layer address Z).

[0072] Figure 3 The "second mapping set" describes the correspondence between transition layer addresses and data storage addresses. For example, transition layer address X is mapped to data storage address X2, and transition layer address Y is mapped to data storage address Y2.

[0073] Figure 3 The "Third Mapping Relationship Set" on the right describes the correspondence between fingerprint values ​​and transition layer addresses. Fingerprint values ​​are used to uniquely identify data content, helping to achieve data deduplication.

[0074] During a write operation, data fingerprint information can be written together with object data. Figure 3 The fingerprint values ​​Hx, He, and Hj displayed in the data block correspond to data blocks X, E, and J, meaning that the fingerprint value of data block X can be the fingerprint value Hx.

[0075] exist Figure 3 In order to facilitate data block alignment, a minimum disk write width can be defined for the underlying storage device, and space can be reserved within the disk write width to store fingerprint values. If the minimum disk write width is not met due to a small number of I / O requests, the data can be stored in the cache for a short period of time, or zeros can be padded to reach the minimum disk write width to complete the underlying data saving operation.

[0076] pass Figure 3 The mapping relationships and data management strategies shown enable the storage system to effectively manage data writing and storage, especially when using advanced storage features such as deduplication and compression. Figure 3 The mapping relationship management method in the database can optimize storage space and improve data access efficiency.

[0077] It should be noted that, in this embodiment of the application, in order to further improve the efficiency of data operation, a reference count is further configured for the mapping relationship between the transition layer address and the data storage address, which is used to indicate the number of logical unit addresses that have established a mapping relationship with the same transition layer address.

[0078] In an optional implementation, after receiving the data write request, the method further includes:

[0079] S1, Obtain data fingerprint information that matches the object data;

[0080] S2-1, if the third mapping relationship set includes a fourth mapping relationship that matches the data fingerprint information, obtain the reference transition layer address according to the fourth mapping relationship; add a fifth mapping relationship between the logical unit address and the reference transition layer address to the first mapping relationship set; search for a sixth mapping relationship in the second mapping relationship set according to the reference transition layer address, wherein the sixth mapping relationship is a mapping relationship between the reference transition layer address and the reference storage address; increase the reference count matching the sixth mapping relationship from a first value to a second value, wherein the reference count is used to indicate the number of logical mapping relationships associated with the reference transition layer address;

[0081] S2-2, If the third mapping relationship set does not include a mapping relationship that matches the data fingerprint information, determine to perform a write operation based on the object data.

[0082] The following combination Figure 4 This section describes a complete data writing process.

[0083] like Figure 4 In the process, S402 is executed first, where the server requests new data to be written.

[0084] Next, step S406 is executed to calculate the fingerprint value of the newly written data. The method for calculating the fingerprint value in this step can be implemented in the same way as in the above-described implementation method.

[0085] Next, execute S408: Query the transition layer address based on the fingerprint value. It can be understood that querying the transition layer address based on the fingerprint value can specifically be done within the aforementioned third mapping relationship set.

[0086] If a record is found, execute S410-1 to S410-3, using the transition layer address in the record (i.e., the fourth mapping relationship); S410-2: add a record (i.e., the fifth mapping relationship) from the logical unit address to the transition layer address, and update the reference count (i.e., the reference count corresponding to the sixth mapping relationship); complete the write request operation.

[0087] If no record is found, execute steps S412-1 to S412-8 to save the data and fingerprint information; wait for the data storage address to be returned; allocate an idle transition layer address; add a record from the transition layer address to the data storage address (i.e., the second mapping relationship) and set the reference count to 1; add a record from the fingerprint value to the transition layer address (i.e., the third mapping relationship); add a record from the logical unit address to the transition layer address (i.e., the first mapping relationship); and complete the write request operation.

[0088] The above-described embodiments of this application optimize data write operations in the storage system. For example, data fingerprint values ​​can be used to improve storage efficiency and reduce data redundancy. During the data write process, data deduplication is achieved by calculating the fingerprint value of the newly written data and checking whether there is a record with the same fingerprint value in the "correspondence between fingerprint value and transition layer address". If the same fingerprint value is found, it means that the data already exists in the storage system and does not need to be written again, thus saving storage space. For duplicate data, the system will not store the same data block repeatedly on the physical storage medium, but will reference the existing data block through a mapping relationship. This method reduces wear and tear on the storage medium and improves the efficiency of the storage system. In addition, when duplicate data is found, the system only needs to add a record to the "correspondence between logical unit address and transition layer address" to complete the write request operation. This operation is much faster than actually writing data to the storage medium, thereby improving the performance of the write operation.

[0089] Furthermore, in the above-described embodiments of this application, when a new record is added to the "correspondence between transition layer address and data storage address", the reference count is set to 1. This helps to track the usage of data blocks and provides a basis for subsequent data management (such as garbage collection).

[0090] In one alternative implementation, when the server data operation request is a data read request, such as Figure 3 In the process, the logical unit address specified by the server can be obtained from the data read request. The current storage device first uses the logical unit address as the key to query the first mapping relationship set, namely "the correspondence between logical unit address and transition layer address", to obtain the transition layer address. Then, using the transition layer address as the key, it queries the second mapping relationship set, "the correspondence between transition layer address and data storage address", to obtain the data storage address. Finally, the data is obtained from the data storage address to complete the read request operation.

[0091] The following describes the garbage collection operation performed based on the above three-layer address structure. In an optional implementation, after adding the second mapping relationship between the transition layer address and the data storage address to the second mapping relationship set, it further includes:

[0092] S1, in response to the storage space reclamation command, determines the first storage address of the current object data to be operated on, and the second storage address after it is moved;

[0093] S2, based on the first storage address, search for the seventh mapping relationship in the second mapping relationship set, wherein the seventh mapping relationship is the mapping relationship between the first storage address and the first transition layer address;

[0094] S3. Based on the second storage address, update the seventh mapping relationship to the eighth mapping relationship, where the eighth mapping relationship is the mapping relationship between the second storage address and the first transition layer address.

[0095] In this application embodiment, a garbage collection method based on a three-layer address scheme is provided. By adding a transition layer address, metadata management is optimized, thereby improving the efficiency of garbage collection and the overall performance of the storage system.

[0096] First, upon receiving a storage space reclamation command, garbage collection is triggered. The garbage collection function in current storage devices periodically cleans up invalid data blocks in the background to free up flash memory space. Because flash memory is characterized by a relatively large granularity in the eraseable blocks that can be reclaimed in a single clearing operation—far larger than the smallest granularity operable in a single read / write operation—garbage collection is necessary to reduce idle flash memory space and quickly reclaim it. To reduce idle flash memory space and quickly reclaim it, typically, some valid data from multiple eraseable blocks is merged and written into a new block, then these multiple eraseable blocks are cleared and reclaimed. After this operation, the original data storage address changes.

[0097] In the above embodiments of this application, by adding a transition layer address, when the background garbage collection function moves data, only the mapping relationship between the transition layer address and the data storage address needs to be updated and adjusted. The mapping relationship between logical unit addresses (i.e., the mapping relationship between logical unit addresses and transition layer addresses) and fingerprint values ​​(i.e., the mapping relationship between fingerprint values ​​and transition layer addresses) does not need to be modified. This avoids the use of a reverse one-to-many relationship mapping table from data storage address to logical unit address, thereby simplifying metadata management during the garbage collection process.

[0098] In an optional implementation, the data deletion operation performed on the above-described three-layer address management scheme can also significantly improve operational efficiency. Specifically, after increasing the reference count matching the sixth mapping relationship from the first value to the second value, the method further includes:

[0099] S1, in response to the data deletion request, based on the first logical unit address carried in the data deletion request, search for the ninth mapping relationship that matches the data deletion request in the first mapping set;

[0100] S2, if the second transition layer address indicated by the ninth mapping relationship matches the reference transition layer address, search for the sixth mapping relationship in the second mapping set according to the reference transition layer address;

[0101] S3, reduce the reference count that matches the sixth mapping relationship from the second value to the third value;

[0102] S4, if the third value is less than or equal to the target threshold, mark the data object corresponding to the reference storage address as a deletable data object according to the sixth mapping relationship.

[0103] Furthermore, after marking the data object corresponding to the reference storage address as a deletable data object according to the sixth mapping relationship when the third value is less than or equal to the target threshold, the above also includes:

[0104] S1, Obtain the data fingerprint information that matches the deletable data object based on the reference storage address;

[0105] S2, based on the data fingerprint information, search for the fourth mapping relationship in the third mapping relationship set;

[0106] S3, based on the reference transition layer address included in the fourth mapping relationship, search for the fifth mapping relationship in the first mapping relationship set;

[0107] S4, delete the fourth mapping relation in the third mapping relation set, and delete the fifth mapping relation in the first mapping relation set.

[0108] Understandably, in the above implementation, when the server initiates a block data deletion request to the storage device by specifying a logical unit address, the storage device can use the logical unit address as the key to look up the "correspondence between logical unit address and transition layer address" and delete the found record. Then, it uses the transition layer address in this record as the key to look up the "correspondence between transition layer address and data storage address." After finding the correspondence, it first decrements the reference count of this record by 1. If the reference count is still greater than zero, the data deletion request operation is completed. If the reference count is zero, the record is deleted, and the data block associated with the data storage address is marked as invalid data awaiting garbage collection. The fingerprint value corresponding to the data is read from the disk width where the data is located and used as the key to look up the "correspondence between fingerprint value and transition layer address" and delete the found record. The garbage collection module is then notified to complete the data deletion request operation.

[0109] By utilizing the above-described embodiments of this application and storing fingerprint values ​​along with other data, the use of mapping tables from data storage addresses to fingerprint values ​​is reduced. This reduction in the use of relational mapping tables decreases the consumption of additional metadata storage space. Furthermore, by adding a transitional layer address, excessive reverse lookups and modifications to mapping relationships by the background garbage collection function are avoided, thereby reducing conflicts arising from concurrent operations of background programs on foreground I / O requests.

[0110] In one alternative implementation, when the server initiates a write request to the storage device, the storage device first deletes the old data and then executes the write request for the new data. This process appears as a single request to the server externally, but internally it is broken down into two independent requests. Specifically, the old data deletion operation may include the following steps:

[0111] First, the storage device finds the corresponding old data storage address based on the logical unit address, and then finds the corresponding transition layer address in the "correspondence between logical unit address and transition layer address".

[0112] Next, the old data storage address is found in the "correspondence between transition layer address and data storage address" using the transition layer address, and the deletion operation is performed.

[0113] A new data write operation may include the following steps:

[0114] The storage device calculates the fingerprint value of the newly written data and uses the fingerprint value as the key to look up the "fingerprint value to transition layer address mapping".

[0115] If a record is found, it means that the data already exists. Directly use the transition layer address in the record to add a new record in the "Correspondence between Logical Unit Address and Transition Layer Address".

[0116] If no record is found, the new data and fingerprint value are handed over to the underlying storage device for storage, and a new data storage address is awaited.

[0117] Then, allocate a free transition layer address in the transition layer address management module, add a new record in the "Correspondence between transition layer address and data storage address", and set the reference count to one.

[0118] Finally, add a new record to both the "correspondence between fingerprint value and transition layer address" and the "correspondence between logical unit address and transition layer address" to complete the write request operation.

[0119] This three-layer addressing scheme reduces the use of mapping tables from data storage addresses to fingerprint values, thereby reducing the storage space consumption of metadata. Simultaneously, by adding a transitional address layer, frequent modifications to the logical unit address mapping table and fingerprint value mapping table are avoided when the background garbage collection function moves data. This reduces concurrent operation conflicts between background programs and foreground I / O requests, improving system stability and efficiency.

[0120] Through the above-described embodiments of this application, the implementation of this application utilizes a three-layer address scheme, which reduces the types of mapping relationship tables, saves storage space occupied by metadata management, and solves the problem of complex software design and difficulty in operation caused by the one-to-many relationship in the traditional metadata management "data storage address to logical unit address correspondence". Compared with the traditional metadata management background garbage collection function moving data, fewer mapping relationship tables need to be modified, thereby reducing the complexity of system processing.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0122] like Figure 5 As shown, embodiments of this application also provide an object data processing apparatus, including:

[0123] The receiving unit 502 is used to receive a data write request, wherein the data write request carries a logical unit address and object data to be written;

[0124] The acquisition unit 504 is used to acquire the data storage address and transition layer address allocated to the object data when the write operation is completed based on the object data. The data storage address is used to indicate the write position of the object data, and the transition layer address determines the mapping relationship between the logical unit address of the object data and the transition layer address.

[0125] The first adding unit 506 is used to add a first mapping relationship between the logical unit address and the transition layer address in the first mapping relationship set;

[0126] The second adding unit 508 is used to add a second mapping relationship between the transition layer address and the data storage address in the second mapping relationship set.

[0127] Optionally, the above-mentioned object data processing apparatus further includes: a third adding unit, used to determine data fingerprint information matching the above-mentioned object data; and when the above-mentioned object data has been written, to add a third mapping relationship between the above-mentioned data fingerprint information and the above-mentioned transition layer address to a third mapping relationship set.

[0128] Optionally, the above-mentioned object data processing apparatus further includes: a first writing unit, configured to acquire the data fingerprint information matching the object data; if the third mapping relationship set includes a fourth mapping relationship matching the data fingerprint information, acquire a reference transition layer address according to the fourth mapping relationship; add a fifth mapping relationship between the logical unit address and the reference transition layer address to the first mapping relationship set; search for a sixth mapping relationship in the second mapping relationship set according to the reference transition layer address, wherein the sixth mapping relationship is a mapping relationship between the reference transition layer address and a reference storage address; increase the reference count matching the sixth mapping relationship from a first value to a second value, wherein the reference count is used to indicate the number of logical mapping relationships associated with the reference transition layer address; and determine to perform the write operation based on the object data if the third mapping relationship set does not include a mapping relationship matching the data fingerprint information.

[0129] Optionally, the above-mentioned object data processing apparatus further includes: a second writing unit, configured to determine multiple sub-object data based on the object data, wherein the data volume of the sub-object data is a preset size; determine data fingerprint values ​​corresponding to each of the multiple sub-object data, wherein the data fingerprint information matching the object data includes the data fingerprint values ​​corresponding to each of the multiple sub-object data; write the data fingerprint values ​​corresponding to each of the sub-object data into multiple consecutive sub-units in the first storage unit; and write multiple sub-object data into multiple consecutive storage units associated with the first storage unit.

[0130] Optionally, the second writing unit is further configured to: in response to a storage space reclamation instruction, determine a first storage address of the current object data to be operated on, and a second storage address after the data has been moved; search for a seventh mapping relationship in the second mapping relationship set based on the first storage address, wherein the seventh mapping relationship is a mapping relationship between the first storage address and the first transition layer address; and update the seventh mapping relationship to an eighth mapping relationship based on the second storage address, wherein the eighth mapping relationship is a mapping relationship between the second storage address and the first transition layer address.

[0131] Optionally, the first writing unit is further configured to: in response to a data deletion request, search for a ninth mapping relationship matching the data deletion request in the first mapping set based on the first logical unit address carried in the data deletion request; if the second transition layer address indicated by the ninth mapping relationship matches the reference transition layer address, search for the sixth mapping relationship in the second mapping set based on the reference transition layer address; reduce the reference count matching the sixth mapping relationship from the second value to the third value; if the third value is less than or equal to a target threshold, mark the data object corresponding to the reference storage address as a deletable data object based on the sixth mapping relationship.

[0132] Optionally, the first writing unit is further configured to: obtain the data fingerprint information matching the deletable data object according to the reference storage address; search for the fourth mapping relationship in the third mapping relationship set according to the data fingerprint information; search for the fifth mapping relationship in the first mapping relationship set according to the reference transition layer address included in the fourth mapping relationship; delete the fourth mapping relationship in the third mapping relationship set, and delete the fifth mapping relationship in the first mapping relationship set.

[0133] For a description of the features in the embodiment corresponding to the object data processing apparatus described above, please refer to the relevant description in the embodiment corresponding to the object data processing method, which will not be repeated here.

[0134] Embodiments of this application also provide an electronic device, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described embodiments of the object data processing method. This electronic device may be... Figure 1 The terminal device or server shown. This embodiment uses a mobile phone or computer as an example for illustration. Figure 6 As shown, the electronic device includes a memory 602 and a processor 604. The memory 602 stores a computer program, and the processor 604 is configured to execute the steps in any of the above method embodiments via the computer program.

[0135] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0136] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0137] S1, Receive a data write request, wherein the data write request carries a logical unit address and object data to be written;

[0138] S2, when the write operation is completed based on the object data, obtain the data storage address and transition layer address allocated to the object data, wherein the data storage address is used to indicate the write position of the object data, and the transition layer address determines the mapping relationship between the logical unit address of the object data and the transition layer address;

[0139] S3, add the first mapping relationship between the logical unit address and the transition layer address to the first mapping relationship set;

[0140] S4, In the second mapping relationship set, add a second mapping relationship between the transition layer address and the data storage address.

[0141] Alternatively, as those skilled in the art will understand, Figure 6 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 6 This does not limit the structure of the aforementioned electronic devices. For example, the electronic device may also include components that are more... Figure 6 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 6 The different configurations shown.

[0142] The memory 602 can be used to store software programs and modules, such as the program instructions / modules corresponding to the object data processing method and apparatus in this embodiment. The processor 604 executes various functional applications and data processing by running the software programs and modules stored in the memory 602, thereby implementing the object data processing method described above. The memory 602 includes the receiving unit 502, the acquiring unit 504, the first adding unit 506, and the second adding unit 508 in the object data processing apparatus described above. The memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 602 may further include memory remotely located relative to the processor 604, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Further details are omitted in this example.

[0143] Optionally, the transmission device 606 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 606 includes a Network Interface Controller (NIC), which can be connected to other network devices and routers via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 606 is a radio frequency (RF) module, used for wireless communication with the Internet.

[0144] In addition, the aforementioned electronic device also includes: a display 608 for displaying the target page; and a connection bus 610 for connecting the various module components in the aforementioned electronic device.

[0145] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the object data processing method.

[0146] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0147] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described object data processing method embodiments.

[0148] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described object data processing method embodiments.

[0149] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0150] The above provides a detailed description of a method for processing object data provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for processing object data, characterized in that, include: Receive a data write request, wherein the data write request carries a logical unit address and object data to be written; When a write operation is completed based on the object data, the data storage address and transition layer address allocated to the object data are obtained, wherein the data storage address is used to indicate the write position of the object data, and the transition layer address determines the mapping relationship between the logical unit address of the object data and the transition layer address; Add a first mapping relationship between the logical unit address and the transition layer address to the first mapping relationship set; In the second mapping relationship set, add a second mapping relationship between the transition layer address and the data storage address; After receiving the data write request, the method further includes: Determine the data fingerprint information that matches the object data; When the object data has been written, a third mapping relationship between the data fingerprint information and the transition layer address is added to the third mapping relationship set; If the third mapping relationship set includes a fourth mapping relationship that matches the data fingerprint information, the reference transition layer address is obtained according to the fourth mapping relationship; In the first mapping relationship set, add a fifth mapping relationship between the logical unit address and the reference transition layer address; Based on the reference transition layer address, in the second mapping relationship set, a sixth mapping relationship is searched, wherein the sixth mapping relationship is the mapping relationship between the reference transition layer address and the reference storage address; The reference count matching the sixth mapping relationship is increased from a first value to a second value, wherein the reference count is used to indicate the number of logical mapping relationships associated with the reference transition layer address; If the third mapping set does not include a mapping relationship that matches the data fingerprint information, it is determined that the write operation will be performed based on the object data.

2. The method according to claim 1, characterized in that, Before obtaining the data storage address and transition layer address allocated to the object data after completing the write operation based on the object data, the method further includes: Multiple sub-object data are determined based on the object data, wherein the data volume of the sub-object data is a preset size; Determine the data fingerprint value corresponding to each of the multiple sub-object data, wherein the data fingerprint information matching the object data includes the data fingerprint value corresponding to each of the multiple sub-object data; In multiple consecutive sub-units within the first storage unit, the data fingerprint value corresponding to each of the sub-object data is written respectively; Multiple sub-object data are written into multiple consecutive storage units associated with the first storage unit.

3. The method according to claim 2, characterized in that, After adding the second mapping relationship between the transition layer address and the data storage address to the second mapping relationship set, the method further includes: In response to a storage space reclamation command, determine the first storage address of the current object data to be operated on, and the second storage address after it is moved; Based on the first storage address, a seventh mapping relationship is searched in the second mapping relationship set, wherein the seventh mapping relationship is the mapping relationship between the first storage address and the first transition layer address; Based on the second storage address, the seventh mapping relationship is updated to the eighth mapping relationship, wherein the eighth mapping relationship is the mapping relationship between the second storage address and the first transition layer address.

4. The method according to claim 1, characterized in that, After incrementing the reference count matching the sixth mapping relationship from the first value to the second value, the method further includes: In response to a data deletion request, based on the first logical unit address carried in the data deletion request, a ninth mapping relationship matching the data deletion request is searched in the first mapping set; If the second transition layer address indicated by the ninth mapping relationship matches the reference transition layer address, the sixth mapping relationship is searched in the second mapping set according to the reference transition layer address; The reference count that matches the sixth mapping relationship is reduced from the second value to the third value; If the third value is less than or equal to the target threshold, the data object corresponding to the reference storage address is marked as a deletable data object according to the sixth mapping relationship.

5. The method according to claim 4, characterized in that, When the third value is less than or equal to the target threshold, after marking the data object corresponding to the reference storage address as a deletable data object according to the fifth mapping relationship, the method further includes: Obtain the data fingerprint information that matches the deletable data object based on the reference storage address; Based on the data fingerprint information, the fourth mapping relationship is searched in the third mapping relationship set; Based on the reference transition layer address included in the fourth mapping relationship, the fifth mapping relationship is searched in the first mapping relationship set; Delete the fourth mapping relationship from the third mapping relationship set, and delete the fifth mapping relationship from the first mapping relationship set.

6. An apparatus for processing object data, characterized in that, include: A receiving unit is used to receive a data write request, wherein the data write request carries a logical unit address and object data to be written; The acquisition unit is used to acquire the data storage address and the transition layer address allocated to the object data when the write operation is completed based on the object data, wherein the data storage address is used to indicate the write position of the object data, and the transition layer address determines the mapping relationship between the logical unit address and the transition layer address of the object data; The first adding unit is used to add a first mapping relationship between the logical unit address and the transition layer address to the first mapping relationship set; The second adding unit is used to add a second mapping relationship between the transition layer address and the data storage address to the second mapping relationship set; The device is further configured to: after receiving the data write request, determine data fingerprint information that matches the object data; and when the object data is written, add a third mapping relationship between the data fingerprint information and the transition layer address to the third mapping relationship set. If the third mapping relationship set includes a fourth mapping relationship that matches the data fingerprint information, the reference transition layer address is obtained according to the fourth mapping relationship; In the first mapping relationship set, add a fifth mapping relationship between the logical unit address and the reference transition layer address; Based on the reference transition layer address, in the second mapping relationship set, a sixth mapping relationship is searched, wherein the sixth mapping relationship is the mapping relationship between the reference transition layer address and the reference storage address; The reference count matching the sixth mapping relationship is increased from a first value to a second value, wherein the reference count is used to indicate the number of logical mapping relationships associated with the reference transition layer address; If the third mapping set does not include a mapping relationship that matches the data fingerprint information, it is determined that the write operation will be performed based on the object data.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 5.

Images