Data writing method and apparatus, communication device, and storage medium

Abstract

This invention provides a data writing method, apparatus, communication device, and storage medium. The method includes: if the data to be written in a first logical data block is not duplicate data and no physical data block has been allocated on the node to which the first logical data block belongs, then setting up a first physical data block on the node to which the first logical data block belongs, and writing the data to be written in the first logical data block to the first physical data block; storing the fingerprint value of the data to be written in the first logical data block and the reference relationship between the first logical data block and the first physical data block in a fingerprint table. This invention solves the problem of large latency and high network bandwidth consumption caused by cross-node data transmission by setting up a first physical data block on the node to which the first logical data block belongs when the data to be written in the first logical data block is not duplicate data and no physical data block has been allocated on the node to which the first logical data block belongs.

Description

Technical Field

[0001] This invention relates to the field of distributed storage system technology, and in particular to a data writing method, apparatus, communication device, and storage medium. Background Technology

[0002] In today's information age, data volume is expanding at an unprecedented rate, not only at the individual user level but also significantly in the data management of enterprises and internet service providers. Distributed storage systems, as a key technology for addressing the demands of massive data storage and efficient access, are becoming increasingly important. However, with the expansion of distributed storage systems, optimizing data access efficiency, especially handling multiple copies of duplicate data, has become one of the bottlenecks in improving the performance of distributed storage systems.

[0003] Existing data writing methods often fail to adequately consider data management, resulting in suboptimal resource utilization during data access. For example, when a user requests access to certain data, the distributed storage system retrieves the data by accessing physical data blocks on other nodes across nodes, rather than prioritizing retrieval from the local node or neighboring nodes. Cross-node data transmission not only increases data transmission latency but also consumes more network bandwidth. Therefore, existing data writing methods suffer from significant latency and high network bandwidth consumption. Summary of the Invention

[0004] In view of this, the present invention aims to provide a data writing method, apparatus, device and storage medium to solve the problems of large latency and large network bandwidth consumption caused by cross-node data transmission.

[0005] According to a first aspect of the present invention, a data writing method is provided, applied to a distributed storage system, wherein the distributed storage system has a plurality of nodes, and each node has a physical data block; the method includes:

[0006] A fingerprint table is pre-configured, wherein the fingerprint table includes fingerprint values ​​corresponding to data in the distributed storage system;

[0007] The data to be written is divided into several logical data blocks according to the logical block address of the data to be written.

[0008] The logical data blocks are allocated to different nodes in the distributed storage system;

[0009] Take any of the aforementioned logical data blocks as the first logical data block, and determine the fingerprint value of the data to be written in the first logical data block;

[0010] Based on the fingerprint table and the fingerprint value of the data to be written in the first logical data block, determine whether the data to be written in the first logical data block is duplicate data.

[0011] If the data to be written in the first logical data block is not duplicate data and the physical data block is not allocated on the node to which the first logical data block belongs, then a first physical data block is set on the node to which the first logical data block belongs, and the data to be written in the first logical data block is written to the first physical data block.

[0012] Establish a reference relationship between the first logical data block and the first physical data block;

[0013] The fingerprint values ​​of the data to be written in the first logical data block and the reference relationship between the first logical data block and the first physical data block are stored in the fingerprint table.

[0014] Optionally, the method further includes:

[0015] Take any of the aforementioned logical data blocks as the second logical data block, and determine the fingerprint value of the data to be written in the second logical data block, wherein the nodes to which the second logical data block and the first logical data block belong are different;

[0016] Based on the fingerprint table and the fingerprint value of the data to be written in the second logical data block, determine whether the data to be written in the second logical data block is duplicate data;

[0017] If the data to be written in the second logical data block is duplicate data, then determine the physical data block that stores the data to be written in the second logical data block;

[0018] If the physical data block storing the data to be written in the second logical data block is the first physical data block, then obtain the reference count of the first physical data block;

[0019] If the number of references to the first physical data block does not exceed a preset threshold, the reference relationship between the first logical data block and the first physical data block in the fingerprint table is updated to the reference relationship between the first logical data block, the second logical data block, and the first physical data block.

[0020] Optionally, after the step of determining the physical data block of the data to be written in the second logical data block, the method further includes:

[0021] If the physical data block storing the data to be written in the second logical data block is the first physical data block and there are multiple first physical data blocks, then obtain the reference counts of multiple first physical data blocks;

[0022] If the number of references to multiple first physical data blocks does not exceed the preset threshold, then the minimum number of references is determined from the number of references to multiple first physical data blocks;

[0023] The first physical data block corresponding to the minimum number of references is taken as the first target physical data block;

[0024] Establish a reference relationship between the second logical data block and the first target physical data block;

[0025] The reference relationship between the second logical data block and the first target physical data block is stored in the fingerprint table.

[0026] Optionally, the method includes:

[0027] Take any of the aforementioned logical data blocks as a third logical data block, and determine the fingerprint value of the data to be written in the third logical data block, wherein the third logical data block and the first logical data block belong to different nodes;

[0028] Based on the fingerprint table and the fingerprint value of the data to be written in the third logical data block, determine whether the data to be written in the third logical data block is duplicate data;

[0029] If the data to be written in the third logical data block is duplicate data, then determine the physical data block that stores the data to be written in the third logical data block;

[0030] If the physical data block storing the data to be written in the third logical data block is the first physical data block, then obtain the reference count of the first physical data block;

[0031] If the number of references to the first physical data block exceeds a preset threshold, then a second physical data block is set on the node to which the third logical data block belongs;

[0032] Establish a reference relationship between the third logical data block and the second physical data block;

[0033] The reference relationship between the third logical data block and the second physical data block is stored in the fingerprint table.

[0034] Optionally, after storing the reference relationship between the third logical data block and the second physical data block in the fingerprint table, the method includes:

[0035] If the third logical data block and the second logical data block belong to the same node, the reference relationship between the third logical data block and the second physical data block in the fingerprint table is updated to the reference relationship between the second logical data block and the second physical data block.

[0036] Update the reference relationship between the first logical data block, the second logical data block and the first physical data block to the reference relationship between the first logical data block and the first physical data block.

[0037] Optionally, the method further includes:

[0038] Take any of the aforementioned logical data blocks as the fourth logical data block, and determine the fingerprint value of the data to be written in the fourth logical data block, wherein a third physical data block is set on the node to which the fourth logical data block belongs.

[0039] Based on the fingerprint table and the fingerprint value of the data to be written in the fourth logical data block, determine whether the data to be written in the fourth logical data block is duplicate data.

[0040] If the data to be written in the fourth logical data block is duplicate data, then determine the physical data block that stores the data to be written in the fourth logical data block;

[0041] If the physical data block storing the data to be written in the fourth logical data block is the third physical data block, then obtain the reference count of the third physical data block;

[0042] If the number of references to the third physical data block does not exceed a preset threshold, then a reference relationship is established between the fourth logical data block and the third physical data block;

[0043] The reference relationship between the fourth logical data block and the third physical data block is stored in the fingerprint table;

[0044] If the physical data block storing the data to be written in the fourth logical data block is the third physical data block and there are multiple third physical data blocks, then obtain the reference count of multiple third physical data blocks;

[0045] If the number of references to multiple third physical data blocks does not exceed the preset threshold, then the minimum number of references is determined from the number of references to multiple physical data blocks;

[0046] The third physical data block corresponding to the minimum number of references is used as the second target physical data block;

[0047] Establish a reference relationship between the fourth logical data block and the second target physical data block;

[0048] The reference relationship between the fourth logical data block and the second target physical data block is stored in the fingerprint table.

[0049] Optionally, after the step of obtaining the reference counts of the plurality of the third physical data blocks, the method includes:

[0050] If the number of references to multiple third physical data blocks exceeds the preset threshold, then physical data blocks storing the data to be written in the fourth logical data block are determined from nodes other than the node to which the fourth logical data block belongs.

[0051] If the physical data blocks on the other nodes store the data to be written in the fourth logical data block and there are multiple physical data blocks on the other nodes, then obtain the reference count of the multiple physical data blocks on the other nodes;

[0052] If the number of references to multiple physical data blocks on the other nodes does not exceed the preset threshold, then the minimum number of references is determined from the number of references to multiple physical data blocks on the other nodes;

[0053] The physical data block corresponding to the minimum number of references is used as the third target physical data block;

[0054] Establish a reference relationship between the fourth logical data block and the third target physical data block;

[0055] The reference relationship between the fourth logical data block and the third target physical data block is stored in the fingerprint table.

[0056] According to a second aspect of the present invention, a data writing device is provided, wherein the distributed storage system is provided with a plurality of nodes, and physical data blocks are provided on the nodes, and the data writing device includes:

[0057] A fingerprint table setting module is used to pre-set a fingerprint table, wherein the fingerprint table includes fingerprint values ​​corresponding to data in the distributed storage system;

[0058] The data segmentation module is used to segment the data to be written into several logical data blocks according to the logical block address of the data to be written, wherein the several logical data blocks include a first logical data block.

[0059] A logical data block allocation module is used to allocate the plurality of logical data blocks to different nodes in the distributed storage system;

[0060] The first fingerprint value determination module is used to take any of the logical data blocks as the first logical data block and determine the fingerprint value of the data to be written in the first logical data block.

[0061] The first duplicate data determination module is used to determine whether the data to be written in the first logical data block is duplicate data based on the fingerprint table and the fingerprint value of the data to be written in the first logical data block.

[0062] The data writing module is used to set up a first physical data block on the node to which the first logical data block belongs and write the data to be written in the first logical data block to the first physical data block if the data to be written in the first logical data block is not duplicate data and the physical data block is not allocated on the node to which the first logical data block belongs.

[0063] The first reference relationship establishment module is used to establish the reference relationship between the first logical data block and the first physical data block;

[0064] The first reference relationship storage module is used to store the fingerprint value of the data to be written in the first logical data block and the reference relationship between the first logical data block and the first physical data block into the fingerprint table.

[0065] Optionally, the device further includes:

[0066] The second fingerprint value determination module is used to take any of the logical data blocks as the second logical data block and determine the fingerprint value of the data to be written in the second logical data block, wherein the nodes to which the second logical data block and the first logical data block belong are different.

[0067] The second duplicate data determination module is used to determine whether the data to be written in the second logical data block is duplicate data based on the fingerprint table and the fingerprint value of the data to be written in the second logical data block.

[0068] The first physical data block determination module is used to determine the physical data block storing the data to be written in the second logical data block if the data to be written in the second logical data block is duplicate data.

[0069] The first physical data block reference count acquisition module is used to acquire the reference count of the first physical data block if the physical data block storing the data to be written in the second logical data block is the first physical data block.

[0070] The second reference relationship establishment module is used to update the reference relationship between the first logical data block and the first physical data block in the fingerprint table to the reference relationship between the first logical data block, the second logical data block and the first physical data block if the number of references to the first physical data block does not exceed a preset threshold.

[0071] Optionally, the device further includes:

[0072] The second physical data block reference count acquisition module is used to acquire the reference counts of multiple first physical data blocks if the physical data block storing the data to be written in the second logical data block is the first physical data block and there are multiple first physical data blocks.

[0073] The first minimum reference count determination module is used to determine the minimum reference count from the reference counts of the multiple first physical data blocks if the reference counts of the multiple first physical data blocks do not exceed the preset threshold.

[0074] The first target physical data block determination module is used to determine the first physical data block corresponding to the minimum number of references as the first target physical data block.

[0075] The third reference relationship establishment module is used to establish a reference relationship between the second logical data block and the first target physical data block;

[0076] The second reference relationship storage module is used to store the reference relationship between the second logical data block and the first target physical data block in the fingerprint table.

[0077] Optionally, the device further includes:

[0078] The third fingerprint value determination module is used to take any of the logical data blocks as the third logical data block and determine the fingerprint value of the data to be written in the third logical data block, wherein the third logical data block and the first logical data block belong to different nodes.

[0079] The third duplicate data judgment module is used to determine whether the data to be written in the third logical data block is duplicate data based on the fingerprint table and the fingerprint value of the data to be written in the third logical data block.

[0080] The second physical data block determination module is used to determine the physical data block storing the data to be written in the third logical data block if the data to be written in the third logical data block is duplicate data.

[0081] The third physical data block reference count acquisition module is used to acquire the reference count of the first physical data block if the physical data block storing the data to be written in the third logical data block is the first physical data block.

[0082] The first newly added physical data block module is used to set a second physical data block on the node to which the third logical data block belongs if the number of references to the first physical data block exceeds a preset threshold.

[0083] The fourth reference relationship establishment module is used to establish the reference relationship between the third logical data block and the second physical data block;

[0084] The third reference relationship storage module is used to store the reference relationship between the third logical data block and the second physical data block in the fingerprint table.

[0085] Optionally, the device further includes:

[0086] The second reference relationship update module is used to update the reference relationship between the third logical data block and the second physical data block in the fingerprint table to the reference relationship between the second logical data block and the second physical data block when the third logical data block and the second logical data block belong to the same node.

[0087] The third reference relationship update module is used to update the reference relationship between the first logical data block, the second logical data block and the first physical data block to the reference relationship between the first logical data block and the first physical data block.

[0088] Optionally, the device further includes:

[0089] The fourth fingerprint value determination module is used to take any of the aforementioned logical data blocks as the fourth logical data block and determine the fingerprint value of the data to be written in the fourth logical data block, wherein a third physical data block is set on the node to which the fourth logical data block belongs.

[0090] The fourth duplicate data determination module is used to determine whether the data to be written in the fourth logical data block is duplicate data based on the fingerprint table and the fingerprint value of the data to be written in the fourth logical data block.

[0091] The third physical data block determination module is used to determine the physical data block storing the data to be written in the fourth logical data block if the data to be written in the fourth logical data block is duplicate data.

[0092] The fourth physical data block reference count acquisition module is used to acquire the reference count of the third physical data block if the physical data block storing the data to be written in the fourth logical data block is the third physical data block.

[0093] The fifth reference relationship establishment module is used to establish a reference relationship between the fourth logical data block and the third physical data block if the number of references to the third physical data block does not exceed a preset threshold.

[0094] The fourth reference relationship storage module is used to store the reference relationship between the fourth logical data block and the third physical data block in the fingerprint table;

[0095] The fifth physical data block reference count acquisition module is used to acquire the reference count of multiple third physical data blocks if the physical data block storing the data to be written in the fourth logical data block is the third physical data block and there are multiple third physical data blocks.

[0096] The second minimum reference count determination module is used to determine the minimum reference count from the reference counts of the multiple physical data blocks if the reference counts of the multiple third physical data blocks do not exceed the preset threshold.

[0097] The second target physical data block determination module is used to take the third physical data block corresponding to the minimum number of references as the second target physical data block.

[0098] The sixth reference relationship establishment module is used to establish a reference relationship between the fourth logical data block and the second target physical data block;

[0099] The fifth reference relationship storage module is used to store the reference relationship between the fourth logical data block and the second target physical data block in the fingerprint table.

[0100] Optionally, the device further includes:

[0101] The fourth physical data block determination module is used to determine the physical data block storing the data to be written in the fourth logical data block from other nodes besides the node to which the fourth logical data block belongs if the number of references to multiple third physical data blocks exceeds the preset threshold.

[0102] The sixth physical data block reference count acquisition module is used to acquire the reference count of multiple physical data blocks on other nodes if the physical data blocks on other nodes store the data to be written in the fourth logical data block and there are multiple physical data blocks on other nodes.

[0103] The third minimum reference count determination module is used to determine the minimum reference count from the reference counts of multiple physical data blocks on the other nodes if the reference counts of multiple physical data blocks on the other nodes do not exceed the preset threshold.

[0104] The third target physical data block determination module is used to determine the physical data block corresponding to the minimum number of references as the third target physical data block.

[0105] The seventh reference relationship establishment module is used to establish a reference relationship between the fourth logical data block and the third target physical data block;

[0106] The sixth reference relationship storage module is used to store the reference relationship between the fourth logical data block and the third target physical data block in the fingerprint table.

[0107] According to another aspect of the present invention, an electronic device is also provided, comprising:

[0108] processor;

[0109] Memory used to store the processor's executable instructions;

[0110] The processor is configured to execute the instructions to implement the data writing method described above.

[0111] According to another aspect of the present invention, a readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the data writing method described above.

[0112] The data writing method provided in this invention is applied to a distributed storage system. The distributed storage system has several nodes, and each node has a physical data block. A fingerprint table is pre-set, which includes fingerprint values ​​corresponding to data in the distributed storage system. The data to be written is divided into several logical data blocks based on the logical block address of the data to be written. These logical data blocks are then allocated to different nodes in the distributed storage system. Any logical data block is designated as a first logical data block, and the fingerprint value of the data to be written in the first logical data block is determined. Based on the fingerprint table and the fingerprint value of the data to be written in the first logical data block, it is determined whether the data to be written in the first logical data block is duplicate data. If the data to be written in the first logical data block is not duplicate data and no physical data block is allocated on the node to which the first logical data block belongs, a first physical data block is set on the node to which the first logical data block belongs, and the data to be written in the first logical data block is written to the first physical data block. A reference relationship is established between the first logical data block and the first physical data block. The fingerprint value of the data to be written in the first logical data block and the reference relationship between the first logical data block and the first physical data block are stored in the fingerprint table. This invention enables the retrieval of the data to be written from the first logical data block within the node to which the first logical data block belongs, provided that the data to be written in the first logical data block is not duplicate data and no physical data block has been allocated on the node to which the first logical data block belongs. This not only saves the transmission bandwidth consumed by data transmission between nodes, but also eliminates the packet caching and forwarding processing of intermediate switching devices between nodes, reducing the latency of data retrieval and solving the problem of large latency and large network bandwidth consumption caused by cross-node data transmission.

[0113] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0114] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0115] Figure 1 This is one of the flowcharts of a data writing method provided in an embodiment of the present invention;

[0116] Figure 2 This is the second step of a data writing method provided in an embodiment of the present invention;

[0117] Figure 3This is the third step in the flowchart of a data writing method provided in this embodiment of the invention;

[0118] Figure 4 This is the fourth step in the flowchart of a data writing method provided in an embodiment of the present invention;

[0119] Figure 5 This is the fifth step in a flowchart of a data writing method provided in an embodiment of the present invention;

[0120] Figure 6 This is the sixth step in the flowchart of a data writing method provided in this embodiment of the invention;

[0121] Figure 7 This is a device block diagram of a data writing apparatus provided in an embodiment of the present invention;

[0122] Figure 8 This is a schematic diagram of a communication device provided in an embodiment of the present invention. Detailed Implementation

[0123] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details are presented in the various embodiments of the present invention to facilitate a better understanding of this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for ease of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with and referenced by each other without contradiction.

[0124] Reference Figure 1 This diagram illustrates one of the steps of a data writing method provided in an embodiment of the present invention, applied to a distributed storage system. The distributed storage system has several nodes, and each node has a physical data block. The method may include:

[0125] Step 101: Pre-set a fingerprint table, wherein the fingerprint table includes fingerprint values ​​corresponding to data in the distributed storage system.

[0126] It should be noted that, in this embodiment of the invention, in order to subsequently determine whether the data to be written is duplicate data, that is, to determine whether the distributed storage system already stores data identical to the data to be written, a fingerprint table needs to be pre-set. The fingerprint table includes the fingerprint values ​​corresponding to each piece of data already stored in the distributed storage system, as well as the storage location information of the stored data within the distributed storage system.

[0127] Step 102: Divide the data to be written into blocks according to the logical block address of the data to be written, and obtain several logical data blocks.

[0128] Step 103: Distribute several logical data blocks to different nodes in the distributed storage system.

[0129] It should be noted that, in this embodiment of the invention, the LBAs (logical block addresses) of the data to be written are obtained in advance. Based on the logical block addresses of the data to be written, the data is divided into blocks to obtain several LDBs (logical data blocks). After obtaining several logical data blocks, these blocks are allocated to different nodes in the distributed storage system.

[0130] Step 104: Take any logical data block as the first logical data block and determine the fingerprint value of the data to be written in the first logical data block.

[0131] Step 105: Based on the fingerprint table and the fingerprint value of the data to be written in the first logical data block, determine whether the data to be written in the first logical data block is duplicate data.

[0132] It should be noted that, in this embodiment of the invention, any one of the several logical data blocks is taken as the first logical data block, and the fingerprint value of the data to be written in the first logical data block is calculated. After obtaining the fingerprint value of the data to be written in the first logical data block, a query is performed in the fingerprint table to see if there is a fingerprint value that is the same as the fingerprint value of the data to be written in the first logical data block. If there is no fingerprint value in the fingerprint table that is the same as the fingerprint value of the data to be written in the first logical data block, it means that the distributed storage system does not store data that is the same as the data to be written in the first logical data block, and the data to be written in the first logical data block is being written to the distributed storage system for the first time. Therefore, it is determined that the data to be written in the first logical data block is not duplicate data. If there is a fingerprint value in the fingerprint table that is the same as the fingerprint value of the data to be written in the first logical data block, it means that the distributed storage system already stores data that is the same as the data to be written in the first logical data block, and the data to be written in the first logical data block is not being written to the distributed storage system for the first time. Therefore, it is determined that the data to be written in the first logical data block is duplicate data.

[0133] Step 106: If the data to be written in the first logical data block is not duplicate data and no physical data block is allocated on the node to which the first logical data block belongs, then set up the first physical data block on the node to which the first logical data block belongs, and write the data to be written in the first logical data block to the first physical data block.

[0134] It should be noted that, in this embodiment of the invention, if the data to be written in the first logical data block is not duplicate data and no physical data block has been allocated on the node to which the first logical data block belongs, then a new PDB (physical data block) is added on the node to which the first logical data block belongs. The newly added physical data block is the first physical data block. After adding the first physical data block, the data to be written in the first logical data block is written into the first physical data block.

[0135] Step 107: Establish the reference relationship between the first logical data block and the first physical data block.

[0136] Step 108: Store the fingerprint value of the data to be written in the first logical data block and the reference relationship between the first logical data block and the first physical data block into the fingerprint table.

[0137] It should be noted that, in this embodiment of the invention, after writing the data to be written from the first logical data block to the first physical data block, a reference relationship is established between the first logical data block and the first physical data block, and the fingerprint value of the data to be written in the first logical data block and the reference relationship between the first logical data block and the first physical data block are stored in a fingerprint table. For example, if the first logical data block is LDB1, the fingerprint value of the data to be written in the first logical data block is FP, and the first physical data block is PDB1, then the fingerprint value of the data to be written in the first logical data block and the reference relationship between the first logical data block and the first physical data block are (FP, LDB1 -> PDB1).

[0138] This invention enables the retrieval of the data to be written from the first logical data block within the node to which the first logical data block belongs, provided that the data to be written in the first logical data block is not duplicate data and no physical data block has been allocated on the node to which the first logical data block belongs. This not only saves the transmission bandwidth consumed by data transmission between nodes, but also eliminates the packet caching and forwarding processing of intermediate switching devices between nodes, reducing the latency of data retrieval and solving the problem of large latency and large network bandwidth consumption caused by cross-node data transmission.

[0139] Furthermore, such as Figure 2 The diagram illustrates a second step of the data writing method provided in this embodiment of the invention. In this embodiment, the method may further include the following steps:

[0140] Step 201: Take any logical data block as the second logical data block, and determine the fingerprint value of the data to be written in the second logical data block, wherein the nodes to which the second logical data block and the first logical data block belong are different.

[0141] Step 202: Based on the fingerprint values ​​of the data to be written in the fingerprint table and the second logical data block, determine whether the data to be written in the second logical data block is duplicate data.

[0142] It should be noted that, in this embodiment of the invention, any one of the several logical data blocks is taken as the second logical data block, and the fingerprint value of the data to be written in the second logical data block is calculated. The nodes to which the second logical data block and the first logical data block belong are different. After obtaining the fingerprint value of the data to be written in the second logical data block, a query is performed in the current fingerprint table to see if there is a fingerprint value identical to the fingerprint value of the data to be written in the second logical data block. If there is no fingerprint value identical to the fingerprint value of the data to be written in the second logical data block in the current fingerprint table, it indicates that the distributed storage system does not store data identical to the data to be written in the second logical data block, and the data to be written in the first logical data block is being written to the distributed storage system for the first time. Therefore, it is determined that the data to be written in the second logical data block is not duplicate data. If there is a fingerprint value identical to the fingerprint value of the data to be written in the second logical data block in the current fingerprint table, it indicates that the distributed storage system already stores data identical to the data to be written in the second logical data block, and the data to be written in the second logical data block is not being written to the distributed storage system for the first time. Therefore, it is determined that the data to be written in the second logical data block is duplicate data.

[0143] Step 203: If the data to be written in the second logical data block is duplicate data, then determine the physical data block that stores the data to be written in the second logical data block.

[0144] Step 204: If the physical data block storing the data to be written in the second logical data block is the first physical data block, then obtain the reference count of the first physical data block.

[0145] Step 205: If the number of references to the first physical data block does not exceed the preset threshold, then update the reference relationship between the first logical data block and the first physical data block in the fingerprint table to the reference relationship between the first logical data block, the second logical data block and the first physical data block.

[0146] It should be noted that, in the embodiments of the present invention, the number of references to the physical data block is related to the number of logical data blocks that reference the physical data block; that is, the number of references to the physical data block is the number of logical data blocks that reference the physical data block.

[0147] If the data to be written in the second logical data block is duplicate data, then a physical data block is determined from the current distributed storage system. It should be noted that the determined physical data block stores the same data as the data to be written in the second logical data block.

[0148] If the identified physical data block is the first physical data block, it means that the first physical data block already stores the same data to be written as the data to be written in the second logical data block. Therefore, it is unnecessary to write the data to be written in the second logical data block to the distributed storage system again. The reference count of the first physical data block is then obtained. It is determined whether the reference count of the first physical data block exceeds a preset threshold. If the reference count of the first physical data block does not exceed the preset threshold, the reference relationship between the first logical data block and the first physical data block in the fingerprint table is updated to a reference relationship between the first logical data block and the second logical data block and the first physical data block, i.e., a new reference relationship is added between the second logical data block and the first physical data block. It should be noted that since the second logical data block and the first logical data block belong to different nodes, and the first physical data block is located on the same node as the first logical data block, the reference relationship between the second logical data block and the first physical data block is a cross-node reference.

[0149] For example: if the first logical data block is LDB1, the fingerprint value of the data to be written in the first logical data block is FP, the first physical data block is PDB1, and the second logical data block is LDB2, then the reference relationship between the first logical data block and the first physical data block in the fingerprint table is updated to the reference relationship between the first logical data block and the second logical data block and the first physical data block, that is, (FP,PDB1<-LDB1) is updated to (FP,PDB1<-(LDB1,LDB2)).

[0150] This invention addresses the issue of duplicate data in a distributed storage system. When the data to be written in the second logical data block is duplicate data, and the physical data block storing this data is the first physical data block, and the reference count of the first physical data block does not exceed a preset threshold, a reference relationship between the second logical data block and the first physical data block is added to the fingerprint table. This avoids a large amount of duplicate data in the distributed storage system and reduces storage space usage. Furthermore, by determining whether to establish a reference relationship based on whether the reference count of the first physical data block exceeds a preset threshold, this invention ensures that duplicate data is not excessively concentrated in the first physical data block, thereby avoiding wasted storage space and optimizing the utilization of storage resources.

[0151] Furthermore, such as Figure 3 The diagram illustrates a third step of the data writing method provided in this embodiment of the invention. In this embodiment, the following steps may be included after step 203:

[0152] Step 301: If the physical data block storing the data to be written in the second logical data block is the first physical data block and there are multiple first physical data blocks, then obtain the reference counts of the multiple first physical data blocks.

[0153] It should be noted that, in this embodiment of the invention, if the physical data block storing the data to be written in the second logical data block is a first physical data block and there are multiple first physical data blocks, then the reference counts corresponding to each of the multiple first physical data blocks are obtained. For the reference count of any first physical data block, the reference count of the first physical data block is compared with a preset threshold to determine whether the reference count of the first physical data block exceeds the preset threshold, thereby determining whether the reference counts of multiple first physical data blocks exceed the preset threshold.

[0154] Step 302: If the number of references to multiple first physical data blocks does not exceed a preset threshold, then determine the minimum number of references from the number of references to multiple first physical data blocks.

[0155] Step 303: Select the first physical data block corresponding to the minimum number of references as the first target physical data block.

[0156] Step 304: Establish the reference relationship between the second logical data block and the first target physical data block.

[0157] Step 305: Store the reference relationship between the second logical data block and the first target physical data block in the fingerprint table.

[0158] It should be noted that, in this embodiment of the invention, if the number of references of multiple first physical data blocks does not exceed a preset threshold, the minimum number of references is determined from the references of multiple first physical data blocks, and the first physical data block corresponding to the minimum number of references is taken as the first target physical data block.

[0159] Establish a reference relationship between the second logical data block and the first target physical data block, and store the reference relationship between the second logical data block and the first target physical data block in the fingerprint table.

[0160] This invention, when the physical data block storing the data to be written in the second logical data block is the first physical data block and there are multiple first physical data blocks, selects the first physical data block with the fewest references as the first target physical data block and establishes a reference relationship between the second logical data block and the first target physical data block. That is, by pointing the new reference relationship to the first target physical data block with the fewest references, the distributed storage system can distribute data storage more evenly and improve the utilization efficiency of storage resources.

[0161] Furthermore, such as Figure 4 The diagram illustrates the fourth step of a data writing method provided in an embodiment of the present invention. In this embodiment, the method may further include the following steps:

[0162] Step 401: Take any logical data block as the third logical data block, and determine the fingerprint value of the data to be written in the third logical data block, wherein the nodes to which the third logical data block and the first logical data block belong are different.

[0163] Step 402: Based on the fingerprint table and the fingerprint value of the data to be written in the third logical data block, determine whether the data to be written in the third logical data block is duplicate data.

[0164] It should be noted that, in this embodiment of the invention, any one of the several logical data blocks is taken as the third logical data block, and the fingerprint value of the data to be written in the third logical data block is calculated. The nodes to which the third logical data block and the first logical data block belong are different. After obtaining the fingerprint value of the data to be written in the third logical data block, a query is performed in the current fingerprint table to see if there is a fingerprint value identical to the fingerprint value of the data to be written in the third logical data block. If there is no fingerprint value identical to the fingerprint value of the data to be written in the third logical data block in the current fingerprint table, it indicates that the distributed storage system does not store data identical to the data to be written in the third logical data block, and the data to be written in the third logical data block is being written to the distributed storage system for the first time. Therefore, it is determined that the data to be written in the third logical data block is not duplicate data. If there is a fingerprint value identical to the fingerprint value of the data to be written in the third logical data block in the current fingerprint table, it indicates that the distributed storage system already stores data identical to the data to be written in the third logical data block, and the data to be written in the third logical data block is not being written to the distributed storage system for the first time. Therefore, it is determined that the data to be written in the third logical data block is duplicate data.

[0165] Step 403: If the data to be written in the third logical data block is duplicate data, then determine the physical data block that stores the data to be written in the third logical data block.

[0166] Step 404: If the physical data block storing the data to be written in the third logical data block is the first physical data block, then obtain the reference count of the first physical data block.

[0167] Step 405: If the number of references to the first physical data block exceeds a preset threshold, then set the second physical data block on the node to which the third logical data block belongs.

[0168] Step 406: Establish the reference relationship between the third logical data block and the second physical data block.

[0169] Step 407: Store the reference relationship between the third logical data block and the second physical data block in the fingerprint table.

[0170] It should be noted that, in this embodiment of the invention, if the data to be written in the third logical data block is duplicate data, a physical data block is determined from the current distributed storage system. It should also be noted that the determined physical data block stores the same data as the data to be written in the third logical data block.

[0171] If the identified physical data block is the first physical data block, it means that the first physical data block already stores the same data to be written as the data in the third logical data block. Therefore, it is unnecessary to write the data to be written from the third logical data block to the distributed storage system again. The reference count of the first physical data block is then determined. It is then checked whether the reference count of the first physical data block exceeds a preset threshold. If the reference count exceeds the preset threshold, a new physical data block is added on the node to which the third logical data block belongs. This new physical data block is the second physical data block, and the data to be written from the third logical data block is written to the second physical data block. After writing the data to be written from the third logical data block to the second physical data block, a reference relationship is established between the third logical data block and the second physical data block, and this reference relationship is stored in the fingerprint table. For example: the first logical data block is LDB1, the fingerprint value of the data to be written in the first logical data block is FP, the first physical data block is PDB1, the second logical data block is LDB2, the third logical data block is LDB3, and the second physical data block is PDB2. Then, the reference relationship between the third logical data block and the second physical data block is PDB2 <- LDB3. Since the data stored in the first physical data block is the same as the data to be written in the third logical data block, the fingerprint value of the data stored in the first physical data block is the same as the fingerprint value of the data to be written in the third logical data block. PDB2 <- LDB3 also corresponds to FP, i.e., (FP, PDB2 <- LDB3). Since (FP, PDB1 <- (LDB1, LDB2)), it can be expressed as (FP, PDB1 <- (LDB1, LDB2), PDB2 <- LDB3).

[0172] This invention uses the physical data block storing the data to be written in the third logical data block as the first physical data block. When the number of references to the first physical data block exceeds a preset threshold, a second physical data block is added within the node to which the third logical data block belongs. This enables the subsequent retrieval of the data to be written in the third logical data block within the node to which the third logical data block belongs. This not only saves the transmission bandwidth consumed by data transmission between nodes, but also eliminates packet caching and forwarding processing of intermediate switching devices between nodes, reducing the latency of data retrieval and solving the problem of large latency and large network bandwidth consumption caused by cross-node data transmission.

[0173] Furthermore, such as Figure 5 The diagram illustrates the fifth step of a data writing method provided in an embodiment of the present invention. In this embodiment, the following steps may be included after step 407:

[0174] Step 501: If the third logical data block and the second logical data block belong to the same node, update the reference relationship between the third logical data block and the second physical data block in the fingerprint table to the reference relationship between the second logical data block and the third logical data block and the second physical data block.

[0175] Step 502: Update the reference relationship between the first logical data block, the second logical data block and the first physical data block to the reference relationship between the first logical data block and the first physical data block.

[0176] It should be noted that, in this embodiment of the invention, if the third logical data block and the second logical data block belong to the same node, the reference relationship between the third logical data block and the second physical data block in the current fingerprint table is updated to the reference relationship between the second logical data block and the third logical data block and the second physical data block. After updating the reference relationship between the third logical data block and the second physical data block to the reference relationship between the second logical data block and the third logical data block and the second physical data block, the reference relationship between the first logical data block, the second logical data block and the first physical data block in the current fingerprint table is updated to the reference relationship between the first logical data block and the first physical data block. For example: the first logical data block is LDB1, the fingerprint value of the data to be written in the first logical data block is FP, the first physical data block is PDB1, the second logical data block is LDB2, the third logical data block is LDB3, and the second physical data block is PDB2. Then, updating the reference relationship between the third logical data block and the second physical data block in the current fingerprint table to the reference relationship between the second logical data block and the third logical data block can be represented as (FP, PDB1 <- (LDB1, LDB2), PDB2 <- LDB3). The reference relationship between the first logical data block, the second logical data block and the first physical data block in the current fingerprint table is updated to the reference relationship between the first logical data block and the first physical data block. This can be expressed as updating (FP,PDB1<-(LDB1,LDB2),PDB2<-(LDB2,LDB3)) to (FP,PDB1<-LDB1,PDB2<-(LDB2,LDB3)).

[0177] This invention achieves shared access to data within a priority node by modifying the cross-node reference relationship between the logical data block and the second physical data block to an intra-node reference relationship. This reduces the need for cross-node data transmission, saves valuable transmission bandwidth, and reduces data access latency, thereby improving the efficiency of data access.

[0178] Furthermore, in this embodiment of the invention, the method may further include the following steps:

[0179] Any logical data block is taken as the fourth logical data block, and the fingerprint value of the data to be written in the fourth logical data block is determined. The node to which the fourth logical data block belongs is equipped with a third physical data block.

[0180] Based on the fingerprint table and the fingerprint value of the data to be written in the fourth logical data block, determine whether the data to be written in the fourth logical data block is duplicate data.

[0181] It should be noted that, in this embodiment of the invention, any one of the several logical data blocks is taken as the fourth logical data block, and the fingerprint value of the data to be written in the fourth logical data block is calculated. A third physical data block is configured on the node to which the fourth logical data block belongs. After obtaining the fingerprint value of the data to be written in the fourth logical data block, a query is performed in the current fingerprint table to see if there is a fingerprint value identical to the fingerprint value of the data to be written in the fourth logical data block. If there is no fingerprint value identical to the fingerprint value of the data to be written in the fourth logical data block in the current fingerprint table, it indicates that the distributed storage system does not store data identical to the data to be written in the fourth logical data block, and the data to be written in the fourth logical data block is being written to the distributed storage system for the first time; therefore, it is determined that the data to be written in the fourth logical data block is not duplicate data. If there is a fingerprint value identical to the fingerprint value of the data to be written in the fourth logical data block in the current fingerprint table, it indicates that the distributed storage system already stores data identical to the data to be written in the fourth logical data block, and the data to be written in the fourth logical data block is not being written to the distributed storage system for the first time; therefore, it is determined that the data to be written in the fourth logical data block is duplicate data.

[0182] If the data to be written in the fourth logical data block is duplicate data, then determine the physical data block that stores the data to be written in the fourth logical data block.

[0183] If the physical data block storing the data to be written in the fourth logical data block is the third physical data block, then obtain the reference count of the third physical data block.

[0184] If the number of references to the third physical data block does not exceed the preset threshold, then a reference relationship is established between the fourth logical data block and the third physical data block.

[0185] The reference relationship between the fourth logical data block and the third physical data block is stored in the fingerprint table.

[0186] It should be noted that, in this embodiment of the invention, if the data to be written in the fourth logical data block is duplicate data, a physical data block is determined from the current distributed storage system. It should also be noted that the determined physical data block stores the same data as the data to be written in the fourth logical data block.

[0187] If the identified physical data block is the third physical data block, it means that the third physical data block already stores the same data to be written as the data in the fourth logical data block. Therefore, it is unnecessary to write the data to be written from the fourth logical data block to the distributed storage system again. In this case, the reference count of the third physical data block is obtained. It is then determined whether the reference count of the third physical data block exceeds a preset threshold. If the reference count of the third physical data block does not exceed the preset threshold, a reference relationship is established between the fourth logical data block and the third physical data block. For example, if the fourth logical data block is LDB4 and the third physical data block is PDB3, the reference relationship between the fourth logical data block and the third physical data block is PDB3 <- LDB4.

[0188] After establishing the reference relationship between the fourth logical data block and the third physical data block, the reference relationship between the fourth logical data block and the third physical data block is stored in the current fingerprint table.

[0189] If the physical data block storing the data to be written in the fourth logical data block is the third physical data block and there are multiple third physical data blocks, then obtain the reference counts of multiple third physical data blocks.

[0190] If the number of references to multiple third physical data blocks does not exceed a preset threshold, then the minimum number of references is determined from the number of references to multiple physical data blocks.

[0191] The third physical data block corresponding to the fewest references is used as the second target physical data block.

[0192] Establish a reference relationship between the fourth logical data block and the second target physical data block.

[0193] The reference relationship between the fourth logical data block and the second target physical data block is stored in the fingerprint table.

[0194] It should be noted that, in this embodiment of the invention, if the physical data block storing the data to be written in the fourth logical data block is a third physical data block and there are multiple third physical data blocks, then the reference counts corresponding to each of the multiple third physical data blocks are obtained. For the reference count of any third physical data block, the reference count of the third physical data block is compared with a preset threshold to determine whether the reference count of the third physical data block exceeds the preset threshold, thereby determining whether the reference counts of multiple third physical data blocks all exceed the preset threshold.

[0195] If the number of references to multiple third physical data blocks does not exceed a preset threshold, then the minimum number of references is determined from the reference counts of multiple third physical data blocks, and the third physical data block corresponding to the minimum number of references is taken as the second target physical data block.

[0196] Establish a reference relationship between the fourth logical data block and the second target physical data block, and store the reference relationship between the fourth logical data block and the second target physical data block in the current fingerprint table.

[0197] This invention establishes a reference relationship between the fourth logical data block and the third physical data block when the physical data block storing the data to be written in the fourth logical data block is the third physical data block on the same node as the fourth logical data block, and the reference count of the third physical data block does not exceed a preset threshold. This not only avoids cross-node data transmission, enabling the distributed storage system to process duplicate data more efficiently, but also reduces data access latency and improves the response speed of the distributed storage system. Simultaneously, by avoiding unnecessary cross-node data transmission, the distributed storage system can better maintain data consistency, improving its overall performance. Furthermore, when the physical data block storing the data to be written in the fourth logical data block is the third physical data block, and there are multiple third physical data blocks, this invention designates the third physical data block with the fewest references as the second target physical data block and establishes a reference relationship between the fourth logical data block and the second target physical data block. By pointing the new reference relationship to the second target physical data block with the fewest references, the distributed storage system can distribute data storage more evenly, improving the utilization efficiency of storage resources.

[0198] Furthermore, such as Figure 6 The diagram illustrates a sixth step of the data writing method provided in this embodiment of the invention. In this embodiment, after the step of "obtaining the reference counts of multiple third physical data blocks," the following steps may also be included:

[0199] Step 601: If the number of references to multiple third physical data blocks exceeds a preset threshold, then determine the physical data block storing the data to be written in the fourth logical data block from other nodes besides the node to which the fourth logical data block belongs.

[0200] Step 602: If the physical data blocks on other nodes store the data to be written in the fourth logical data block and there are multiple physical data blocks on other nodes, then obtain the reference counts of the multiple physical data blocks on other nodes.

[0201] Step 603: If the number of references to multiple physical data blocks on other nodes does not exceed a preset threshold, then determine the minimum number of references from the number of references to multiple physical data blocks on other nodes.

[0202] Step 604: Select the physical data block with the fewest references as the third target physical data block.

[0203] Step 605: Establish the reference relationship between the fourth logical data block and the third target physical data block.

[0204] Step 606: Store the reference relationship between the fourth logical data block and the third target physical data block in the fingerprint table.

[0205] It should be noted that, in this embodiment of the invention, if the number of references to multiple third physical data blocks all exceed a preset threshold, then the cross-node method determines from other nodes, excluding the node to which the fourth logical data block belongs, whether there exists a physical data block that stores the same data to be written as the data in the fourth logical data block.

[0206] If there are physical data blocks on other nodes that store the same data to be written as the data in the fourth logical data block, and there are multiple physical data blocks on other nodes, then obtain the reference counts of each of the multiple physical data blocks on the other nodes.

[0207] For any physical data block on other nodes, the reference count of the physical data block is compared with a preset threshold to determine whether the reference count of the physical data block exceeds the preset threshold. In this way, it can be determined whether the reference counts of multiple third physical data blocks on other nodes all exceed the preset threshold.

[0208] If the number of references to multiple third physical data blocks on other nodes does not exceed the preset threshold, then the minimum number of references is determined from the number of references to multiple third physical data blocks on other nodes, and the physical data block corresponding to the minimum number of references is taken as the third target physical data block.

[0209] Establish a reference relationship between the fourth logical data block and the third target physical data block, and store the reference relationship between the fourth logical data block and the third target physical data block in the current fingerprint table.

[0210] This invention avoids performance bottlenecks caused by excessively high reference counts of physical data blocks within a node, and ensures the stable operation of the distributed storage system by referencing the physical data block with the fewest reference counts on other nodes across nodes when the reference counts of physical data blocks within a node all exceed a preset threshold.

[0211] Furthermore, in this embodiment of the invention, the following steps may be included after step 606:

[0212] If the physical data blocks on other nodes do not store the data to be written in the fourth logical data block, or if the number of references to multiple physical data blocks on other nodes exceeds a preset threshold, then the fourth physical data block is set on the node to which the fourth logical data block belongs.

[0213] Establish a reference relationship between the fourth logical data block and the fourth physical data block.

[0214] The reference relationship between the fourth logical data block and the fourth physical data block is stored in the fingerprint table.

[0215] It should be noted that, in this embodiment of the invention, if no physical data block storing the same data as the data to be written in the fourth logical data block is found on other nodes besides the node to which the fourth logical data block belongs, or if multiple physical data blocks on other nodes are found to store the same data as the data to be written in the fourth logical data block and the reference count of the multiple physical data blocks on other nodes exceeds a preset threshold, then a new physical data block is added on the node to which the fourth logical data block belongs, and the newly added physical data block is the fourth physical data block.

[0216] The data to be written from the fourth logical data block is written to the fourth physical data block. After the data to be written from the fourth logical data block to the fourth physical data block is completed, a reference relationship is established between the fourth logical data block and the fourth physical data block. After the reference relationship between the fourth logical data block and the fourth physical data block is established, the reference relationship between the fourth logical data block and the fourth physical data block is stored in the current fingerprint table.

[0217] This invention addresses situations where physical data blocks on other nodes do not store the data to be written in the fourth logical data block, or where the number of references to multiple physical data blocks on other nodes exceeds a preset threshold. In such cases, a fourth physical data block is set on the node to which the fourth logical data block belongs, and a reference relationship is established between the fourth logical data block and the fourth physical data block. This reduces data transmission latency and bandwidth consumption by sacrificing data duplication rate to achieve intra-node referencing.

[0218] According to an embodiment of the present invention, a data writing device is also provided, referring to... Figure 7 , Figure 7 This is a block diagram of a data writing device provided in an embodiment of the present invention. The data writing device includes:

[0219] The fingerprint table setting module 701 is used to pre-set the fingerprint table, wherein the fingerprint table includes fingerprint values ​​corresponding to data in the distributed storage system;

[0220] The data segmentation module 702 is used to segment the data to be written into several logical data blocks according to the logical block address of the data to be written, wherein the several logical data blocks include the first logical data block.

[0221] The logical data block allocation module 703 is used to allocate several logical data blocks to different nodes in a distributed storage system.

[0222] The first fingerprint value determination module 704 is used to take any logical data block as the first logical data block and determine the fingerprint value of the data to be written in the first logical data block.

[0223] The first duplicate data judgment module 705 is used to determine whether the data to be written in the first logical data block is duplicate data based on the fingerprint table and the fingerprint value of the data to be written in the first logical data block.

[0224] The data writing module 706 is used to set up a first physical data block on the node to which the first logical data block belongs if the data to be written in the first logical data block is not duplicate data and no physical data block is allocated on the node to which the first logical data block belongs, and to write the data to be written in the first logical data block to the first physical data block.

[0225] The first reference relationship establishment module 707 is used to establish the reference relationship between the first logical data block and the first physical data block;

[0226] The first reference relationship storage module 708 is used to store the fingerprint value of the data to be written in the first logical data block and the reference relationship between the first logical data block and the first physical data block into the fingerprint table.

[0227] Optionally, the device further includes:

[0228] The second fingerprint value determination module is used to take any logical data block as the second logical data block and determine the fingerprint value of the data to be written in the second logical data block, wherein the nodes to which the second logical data block and the first logical data block belong are different.

[0229] The second duplicate data judgment module is used to determine whether the data to be written in the second logical data block is duplicate data based on the fingerprint value of the data to be written in the fingerprint table and the second logical data block.

[0230] The first physical data block determination module is used to determine the physical data block storing the data to be written in the second logical data block if the data to be written in the second logical data block is duplicate data.

[0231] The first physical data block reference count acquisition module is used to acquire the reference count of the first physical data block if the physical data block storing the data to be written in the second logical data block is the first physical data block.

[0232] The second reference relationship establishment module is used to update the reference relationship between the first logical data block and the first physical data block in the fingerprint table to the reference relationship between the first logical data block, the second logical data block and the first physical data block if the number of references to the first physical data block does not exceed a preset threshold.

[0233] Optionally, the device further includes:

[0234] The second physical data block reference count acquisition module is used to acquire the reference counts of multiple first physical data blocks if the physical data block storing the data to be written in the second logical data block is the first physical data block and there are multiple first physical data blocks.

[0235] The first minimum reference count determination module is used to determine the minimum reference count from the reference counts of multiple first physical data blocks if the reference counts of multiple first physical data blocks do not exceed a preset threshold.

[0236] The first target physical data block determination module is used to determine the first physical data block corresponding to the minimum number of references as the first target physical data block.

[0237] The third reference relationship establishment module is used to establish the reference relationship between the second logical data block and the first target physical data block;

[0238] The second reference relationship storage module is used to store the reference relationship between the second logical data block and the first target physical data block in the fingerprint table.

[0239] Optionally, the device further includes:

[0240] The third fingerprint value determination module is used to take any logical data block as the third logical data block and determine the fingerprint value of the data to be written in the third logical data block, wherein the third logical data block and the first logical data block belong to different nodes.

[0241] The third duplicate data judgment module is used to determine whether the data to be written in the third logical data block is duplicate data based on the fingerprint table and the fingerprint value of the data to be written in the third logical data block.

[0242] The second physical data block determination module is used to determine the physical data block storing the data to be written in the third logical data block if the data to be written in the third logical data block is duplicate data.

[0243] The third physical data block reference count acquisition module is used to acquire the reference count of the first physical data block if the physical data block storing the data to be written in the third logical data block is the first physical data block.

[0244] The first newly added physical data block module is used to set a second physical data block on the node to which the third logical data block belongs if the number of references to the first physical data block exceeds a preset threshold.

[0245] The fourth reference relationship establishment module is used to establish the reference relationship between the third logical data block and the second physical data block;

[0246] The third reference relationship storage module is used to store the reference relationship between the third logical data block and the second physical data block in the fingerprint table.

[0247] Optionally, the device further includes:

[0248] The second reference relationship update module is used to update the reference relationship between the third logical data block and the second physical data block in the fingerprint table to the reference relationship between the second logical data block and the second physical data block when the third logical data block and the second logical data block belong to the same node.

[0249] The third reference relationship update module is used to update the reference relationship between the first logical data block, the second logical data block and the first physical data block to the reference relationship between the first logical data block and the first physical data block.

[0250] Optionally, the device further includes:

[0251] The fourth fingerprint value determination module is used to take any logical data block as the fourth logical data block and determine the fingerprint value of the data to be written in the fourth logical data block. The node to which the fourth logical data block belongs is equipped with a third physical data block.

[0252] The fourth duplicate data judgment module is used to determine whether the data to be written in the fourth logical data block is duplicate data based on the fingerprint table and the fingerprint value of the data to be written in the fourth logical data block.

[0253] The third physical data block determination module is used to determine the physical data block storing the data to be written in the fourth logical data block if the data to be written in the fourth logical data block is duplicate data.

[0254] The fourth physical data block reference count acquisition module is used to acquire the reference count of the third physical data block if the physical data block storing the data to be written in the fourth logical data block is the third physical data block.

[0255] The fifth reference relationship establishment module is used to establish a reference relationship between the fourth logical data block and the third physical data block if the number of references to the third physical data block does not exceed a preset threshold.

[0256] The fourth reference relationship storage module is used to store the reference relationship between the fourth logical data block and the third physical data block in the fingerprint table;

[0257] The fifth physical data block reference count acquisition module is used to acquire the reference count of multiple third physical data blocks if the physical data block storing the data to be written in the fourth logical data block is the third physical data block and there are multiple third physical data blocks.

[0258] The second minimum reference count determination module is used to determine the minimum reference count from the reference counts of multiple physical data blocks if the reference counts of multiple third physical data blocks do not exceed a preset threshold.

[0259] The second target physical data block determination module is used to select the third physical data block corresponding to the minimum number of references as the second target physical data block.

[0260] The sixth reference relationship establishment module is used to establish the reference relationship between the fourth logical data block and the second target physical data block;

[0261] The fifth reference relationship storage module is used to store the reference relationship between the fourth logical data block and the second target physical data block in the fingerprint table.

[0262] Optionally, the device further includes:

[0263] The fourth physical data block determination module is used to determine the physical data block storing the data to be written in the fourth logical data block from other nodes besides the node to which the fourth logical data block belongs if the number of references to multiple third physical data blocks exceeds a preset threshold.

[0264] The sixth physical data block reference count acquisition module is used to acquire the reference count of multiple physical data blocks on other nodes if physical data blocks on other nodes store data to be written in the fourth logical data block and there are multiple physical data blocks on other nodes.

[0265] The third minimum reference count determination module is used to determine the minimum reference count from the reference counts of multiple physical data blocks on other nodes if the reference counts of multiple physical data blocks on other nodes do not exceed a preset threshold.

[0266] The third target physical data block determination module is used to select the physical data block with the fewest references as the third target physical data block.

[0267] The seventh reference relationship establishment module is used to establish the reference relationship between the fourth logical data block and the third target physical data block;

[0268] The sixth reference relationship storage module is used to store the reference relationship between the fourth logical data block and the third target physical data block in the fingerprint table.

[0269] This invention also provides an electronic device, such as... Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, communication interface 802, and memory 803 communicate with each other via the communication bus 804.

[0270] Memory 803 is used to store computer programs;

[0271] When processor 801 executes a program stored in memory 803, it performs the following steps:

[0272] A fingerprint table is pre-configured, which includes fingerprint values ​​corresponding to data in the distributed storage system.

[0273] Based on the logical block address of the data to be written, the data to be written is divided into several logical data blocks;

[0274] Distribute several logical data blocks to different nodes in a distributed storage system;

[0275] Take any logical data block as the first logical data block and determine the fingerprint value of the data to be written in the first logical data block.

[0276] Based on the fingerprint table and the fingerprint value of the data to be written in the first logical data block, determine whether the data to be written in the first logical data block is duplicate data;

[0277] If the data to be written in the first logical data block is not duplicate data and no physical data block is allocated on the node to which the first logical data block belongs, then the first physical data block is set on the node to which the first logical data block belongs, and the data to be written in the first logical data block is written to the first physical data block.

[0278] Establish a reference relationship between the first logical data block and the first physical data block;

[0279] The fingerprint values ​​of the data to be written in the first logical data block and the reference relationship between the first logical data block and the first physical data block are stored in the fingerprint table.

[0280] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0281] The communication interface is used for communication between the aforementioned terminal and other devices.

[0282] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0283] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0284] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the data writing methods described in the above embodiments.

[0285] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive, or SSD).

[0286] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0287] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0288] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data writing method, characterized in that, The method is applied to a distributed storage system, wherein the distributed storage system has several nodes, and each node has physical data blocks configured on it. A fingerprint table is pre-configured, wherein the fingerprint table includes fingerprint values ​​corresponding to data in the distributed storage system; The data to be written is divided into several logical data blocks according to the logical block address of the data to be written. The logical data blocks are allocated to different nodes in the distributed storage system; Take any of the aforementioned logical data blocks as the first logical data block, and determine the fingerprint value of the data to be written in the first logical data block; Based on the fingerprint table and the fingerprint value of the data to be written in the first logical data block, determine whether the data to be written in the first logical data block is duplicate data. If the data to be written in the first logical data block is not duplicate data and the physical data block is not allocated on the node to which the first logical data block belongs, then a first physical data block is set on the node to which the first logical data block belongs, and the data to be written in the first logical data block is written to the first physical data block. Establish a reference relationship between the first logical data block and the first physical data block; The fingerprint values ​​of the data to be written in the first logical data block and the reference relationship between the first logical data block and the first physical data block are stored in the fingerprint table.

2. The method according to claim 1, characterized in that, The method further includes: Take any of the aforementioned logical data blocks as the second logical data block, and determine the fingerprint value of the data to be written in the second logical data block, wherein the nodes to which the second logical data block and the first logical data block belong are different; Based on the fingerprint table and the fingerprint value of the data to be written in the second logical data block, determine whether the data to be written in the second logical data block is duplicate data; If the data to be written in the second logical data block is duplicate data, then determine the physical data block that stores the data to be written in the second logical data block; If the physical data block storing the data to be written in the second logical data block is the first physical data block, then obtain the reference count of the first physical data block; If the number of references to the first physical data block does not exceed a preset threshold, the reference relationship between the first logical data block and the first physical data block in the fingerprint table is updated to the reference relationship between the first logical data block, the second logical data block, and the first physical data block.

3. The method according to claim 2, characterized in that, After the step of determining the physical data block of the data to be written in the second logical data block, the method further includes: If the physical data block storing the data to be written in the second logical data block is the first physical data block and there are multiple first physical data blocks, then obtain the reference counts of multiple first physical data blocks; If the number of references to multiple first physical data blocks does not exceed the preset threshold, then the minimum number of references is determined from the number of references to multiple first physical data blocks; The first physical data block corresponding to the minimum number of references is taken as the first target physical data block; Establish a reference relationship between the second logical data block and the first target physical data block; The reference relationship between the second logical data block and the first target physical data block is stored in the fingerprint table.

4. The method according to claim 1, characterized in that, The method includes: Take any of the aforementioned logical data blocks as a third logical data block, and determine the fingerprint value of the data to be written in the third logical data block, wherein the third logical data block and the first logical data block belong to different nodes; Based on the fingerprint table and the fingerprint value of the data to be written in the third logical data block, determine whether the data to be written in the third logical data block is duplicate data; If the data to be written in the third logical data block is duplicate data, then determine the physical data block that stores the data to be written in the third logical data block; If the physical data block storing the data to be written in the third logical data block is the first physical data block, then obtain the reference count of the first physical data block; If the number of references to the first physical data block exceeds a preset threshold, then a second physical data block is set on the node to which the third logical data block belongs; Establish a reference relationship between the third logical data block and the second physical data block; The reference relationship between the third logical data block and the second physical data block is stored in the fingerprint table.

5. The method according to claim 4, characterized in that, After storing the reference relationship between the third logical data block and the second physical data block in the fingerprint table, the method includes: If the third logical data block and the second logical data block belong to the same node, the reference relationship between the third logical data block and the second physical data block in the fingerprint table is updated to the reference relationship between the second logical data block and the second physical data block. Update the reference relationship between the first logical data block, the second logical data block and the first physical data block to the reference relationship between the first logical data block and the first physical data block.

6. The method according to claim 1, characterized in that, The method further includes: Take any of the aforementioned logical data blocks as the fourth logical data block, and determine the fingerprint value of the data to be written in the fourth logical data block, wherein a third physical data block is set on the node to which the fourth logical data block belongs. Based on the fingerprint table and the fingerprint value of the data to be written in the fourth logical data block, determine whether the data to be written in the fourth logical data block is duplicate data. If the data to be written in the fourth logical data block is duplicate data, then determine the physical data block that stores the data to be written in the fourth logical data block; If the physical data block storing the data to be written in the fourth logical data block is the third physical data block, then obtain the reference count of the third physical data block; If the number of references to the third physical data block does not exceed a preset threshold, then a reference relationship is established between the fourth logical data block and the third physical data block; The reference relationship between the fourth logical data block and the third physical data block is stored in the fingerprint table; If the physical data block storing the data to be written in the fourth logical data block is the third physical data block and there are multiple third physical data blocks, then obtain the reference count of multiple third physical data blocks; If the number of references to multiple third physical data blocks does not exceed the preset threshold, then the minimum number of references is determined from the number of references to multiple physical data blocks; The third physical data block corresponding to the minimum number of references is used as the second target physical data block; Establish a reference relationship between the fourth logical data block and the second target physical data block; The reference relationship between the fourth logical data block and the second target physical data block is stored in the fingerprint table.

7. The method according to claim 6, characterized in that, After the step of obtaining the reference counts of the multiple third physical data blocks, the method includes: If the number of references to multiple third physical data blocks exceeds the preset threshold, then physical data blocks storing the data to be written in the fourth logical data block are determined from nodes other than the node to which the fourth logical data block belongs. If the physical data blocks on the other nodes store the data to be written in the fourth logical data block and there are multiple physical data blocks on the other nodes, then obtain the reference count of the multiple physical data blocks on the other nodes; If the number of references to multiple physical data blocks on the other nodes does not exceed the preset threshold, then the minimum number of references is determined from the number of references to multiple physical data blocks on the other nodes; The physical data block corresponding to the minimum number of references is used as the third target physical data block; Establish a reference relationship between the fourth logical data block and the third target physical data block; The reference relationship between the fourth logical data block and the third target physical data block is stored in the fingerprint table.

8. A data writing device, characterized in that, Applied to a distributed storage system, wherein the distributed storage system has several nodes, and each node has a physical data block, the data writing device includes: A fingerprint table setting module is used to pre-set a fingerprint table, wherein the fingerprint table includes fingerprint values ​​corresponding to data in the distributed storage system; The data segmentation module is used to segment the data to be written into several logical data blocks according to the logical block address of the data to be written, wherein the several logical data blocks include a first logical data block. A logical data block allocation module is used to allocate the plurality of logical data blocks to different nodes in the distributed storage system; The fingerprint value determination module is used to take any of the aforementioned logical data blocks as the first logical data block and determine the fingerprint value of the data to be written in the first logical data block. The duplicate data determination module is used to determine whether the data to be written in the first logical data block is duplicate data based on the fingerprint table and the fingerprint value of the data to be written in the first logical data block. The data writing module is used to set up a first physical data block on the node to which the first logical data block belongs and write the data to be written in the first logical data block to the first physical data block if the data to be written in the first logical data block is not duplicate data and the physical data block is not allocated on the node to which the first logical data block belongs. The reference relationship establishment module is used to establish the reference relationship between the first logical data block and the first physical data block; The reference relationship storage module is used to store the fingerprint value of the data to be written in the first logical data block and the reference relationship between the first logical data block and the first physical data block into the fingerprint table.

9. An electronic device, characterized in that, include: Memory used to store processor-executable instructions; The processor is configured to execute the instructions to implement the data writing method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, A computer program is stored on the readable storage medium, which, when executed by a processor, implements the data writing method as described in any one of claims 1 to 7.

Images