Data writing method and related device

Abstract

The invention discloses a data writing method and a related device, and relates to the technical field of data processing, the method receives an external data writing request, only provides a logic position of to-be-written data, utilizes a metadata server to centrally manage file mapping, prevents each data server from independently maintaining a data copy, and improves the data writing efficiency. And the introduction of the data fragment ID allows the data to be decoupled into manageable fragments, so that the initial data transmission quantity and the size of a transmission unit are reduced. Data to be written are blocked to obtain data blocks to be written, data block names are set for the data blocks to be written, network communication is shared among the data servers, an independent link does not need to be maintained for each data server, the data blocks of the data blocks to be written are only sent to the distributed data servers instead of all the data servers, concurrent repeated data streams are eliminated, and the data processing efficiency is improved. The network congestion risk is reduced, and each data server only stores the required data of the data block to be written, so that the utilization rate of storage resources of the data server is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data writing method and related device. BACKGROUND

[0002] A distributed storage system stores complete data in multiple physically or logically independent data servers to achieve high reliability and scalability.

[0003] At present, a complete copy of the same data is generally sent to each data server through an independent data transmission link, so that each data server stores the data.

[0004] However, this distribution mode needs to establish and maintain an independent data transmission link for each data server. When multiple complete copies of the same data are transmitted to each data server through their respective independent channels, in the case of writing large files or high concurrency, these large amounts of repeated data streams will continuously occupy bandwidth resources, leading to a sharp rise in the risk of network congestion, and each data server needs to store all complete data, resulting in redundant storage content and low storage resource utilization. SUMMARY

[0005] In view of the above problems, the present application provides a data writing method and related device to reduce the risk of network congestion and improve the utilization of storage resources of the data server. The specific scheme is as follows:

[0006] The first aspect of the present application provides a data writing method, which is applied to any data server, and each data server is communicatively connected with each other and with a metadata server. The method comprises the following steps:

[0007] receiving a data writing request, the data writing request carrying a file name of a file where the to-be-written data is located and a logical position of the to-be-written data in the file where the to-be-written data is located;

[0008] sending a query request to the metadata server, the query request carrying the file name of the file where the to-be-written data is located;

[0009] receiving a data shard ID allocated by the metadata server for the to-be-written data, the data shard ID of the to-be-written data being a data shard ID obtained by the metadata server based on the file name of the file where the to-be-written data is located;

[0010] chunking the received to-be-written data to obtain at least one to-be-written data block, and setting a data block name for the to-be-written data block based on a data shard ID of the to-be-written data and a logical position of the to-be-written data block in the to-be-written data;

[0011] obtaining a node quantity of a data server based on a data server list obtained from the metadata server, and allocating at least one data server for the to-be-written data block based on the data block name of the to-be-written data block and the node quantity of the data server;

[0012] sending the to-be-written data block allocated to the local data server to the local data server, or sending the to-be-written data block allocated to another data server to the another data server through the local data server, the another data server being a data server other than the local data server.

[0013] In a possible implementation, after the sending of the to-be-written data block allocated to the local data server to the local data server, or the sending of the to-be-written data block allocated to another data server to the another data server through the local data server, the method further includes:

[0014] sending a query request to the metadata server, the query request carrying a file name of a file in which the to-be-written data is located;

[0015] receiving an inode ID of the file in which the to-be-written data is located returned by the metadata server, the inode ID of the file in which the to-be-written data is located being obtained based on a query of a preset file index by the file name of the file in which the to-be-written data is located, the preset file index including file names and inode IDs of a plurality of files;

[0016] setting a data name for the to-be-written data based on a logical position of the to-be-written data in the file in which the to-be-written data is located and a data shard ID of the to-be-written data;

[0017] sending the inode ID of the file in which the to-be-written data is located and the data name of the to-be-written data to the metadata server, so that the metadata server updates the preset file index based on the inode ID of the file in which the to-be-written data is located and the data name of the to-be-written data.

[0018] In a possible implementation, before the receiving of the data write request, the method further includes:

[0019] Receive a file creation request for the target file, the file creation request carrying the file creation name and file creation path;

[0020] The file creation request is sent to the metadata server so that the metadata server creates the target file based on the target creation path determined by parsing the file creation path, and the name of the target file is the file creation name.

[0021] In one possible implementation, after sending the inode ID of the file containing the data to be written and the data name of the data to be written to the metadata server, the method further includes:

[0022] Send a data read request to the metadata server. The data read request carries the file name of the file containing the data to be read, and the logical position of the data to be read in the file containing the data to be read.

[0023] The metadata server receives the data fragment ID of the stored data and the logical position of the stored data in the file where the data to be read is located, based on the file name of the file where the data to be read is located and the logical position of the data to be read in the file where the data to be read is located.

[0024] Based on the logical position of the data to be read in the file where the data to be read is located, the data fragment ID of the stored data where the data to be read is located, and the logical position of the stored data in the file where the data to be read is located, a data block name is set for the data block to be read;

[0025] Based on the data block name of the data block to be read and the list of data servers obtained from the metadata server, determine the data server where the data block to be read is located.

[0026] The data to be read can be read from the local data server where the data block to be read is located, or the data to be read can be read from another data server where the data block to be read is located through the local data server.

[0027] In one possible implementation, allocating at least one data server for the data block to be written based on the data block name and the number of nodes of the data server includes:

[0028] Based on the data block name of the data block to be written, the hash value of the data block to be written is calculated using a hash algorithm;

[0029] allocating at least one data server for the to-be-written data block based on the hash value of the to-be-written data block and the number of the data servers.

[0030] In a possible implementation, the setting of the data block name for the to-be-read data block based on the logical position of the to-be-read data in the file where the to-be-read data is located, the data shard ID of the stored data where the to-be-read data is located, and the logical position of the stored data in the file where the to-be-read data is located includes:

[0031] calculating the logical position of the to-be-read data block in the to-be-read data based on the logical position of the to-be-read data in the file where the to-be-read data is located and the logical position of the stored data in the file where the to-be-read data is located.

[0032] setting the data block name for the to-be-read data block based on the data shard ID of the stored data where the to-be-read data is located and the logical position of the to-be-read data block in the to-be-read data.

[0033] The second aspect of the present application provides a data writing method, which is applied to a metadata server, the metadata server is in communication connection with a plurality of data servers, and each of the data servers is in communication connection with each other, and the method comprises the following steps.

[0034] receiving a query request sent by the data server after receiving a data writing request, the data writing request carrying a file name of a file where to-be-written data is located and a logical position of the to-be-written data in the file where the to-be-written data is located, and the query request carrying the file name of the file where the to-be-written data is located;

[0035] allocating a data shard ID for the to-be-written data based on the file name of the file where the to-be-written data is located;

[0036] sending the data shard ID of the to-be-written data to the data server, so that the data server performs block division on the received to-be-written data to obtain at least one to-be-written data block;

[0037] sending the data server list to the data servers, so that the data servers allocate at least one of the data servers to the to-be-written data block based on a name of the to-be-written data block and a node number of the data servers, and the data servers send the to-be-written data block allocated to a local data server to the local data server, or send the to-be-written data block allocated to another data server to the another data server through the local data server, wherein the name of the to-be-written data block is set by the data servers based on a data shard ID of the to-be-written data and a logical position of the to-be-written data block in the to-be-written data, the node number of the data servers is obtained based on the data server list, and the another data server is a data server other than the local data server.

[0038] In a possible implementation, the data shard ID is allocated to the to-be-written data based on a file name of a file in which the to-be-written data is located, comprising:

[0039] querying, based on the file name of the file in which the to-be-written data is located, a data shard ID of stored data in a file corresponding to the file name of the file in which the to-be-written data is located in the preset file index;

[0040] allocating a new data shard ID to the to-be-written data based on the data shard ID of the stored data in the file corresponding to the file name of the file in which the to-be-written data is located.

[0041] The third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0042] The memory is configured to store a computer program.

[0043] The processor is configured to execute the computer program, so that the electronic device can implement the data writing method of the first aspect or any implementation manner of the first aspect.

[0044] The fourth aspect of the present application provides a computer storage medium, the storage medium carries one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can implement the data writing method of the first aspect or any implementation manner of the first aspect.

[0045] By means of the technical scheme, the application provides a data writing method and related device. The method is applied to any data server, and each data server is in communication connection with each other and with a metadata server. The method receives a file name of a file in which to-be-written data is located and a logical position of the to-be-written data in the file, sends the file name of the file in which the to-be-written data is located to the metadata server, receives a data shard ID allocated by the metadata server for the to-be-written data based on the file name, and introduces the data shard ID to allow the data to be decoupled into manageable shards, thereby reducing the initial data transmission amount and the size of the transmission unit. The received to-be-written data is divided into at least one to-be-written data block, and a data block name is set for the to-be-written data block based on the data shard ID of the to-be-written data and the logical position of the to-be-written data block in the to-be-written data. At least one data server is allocated for the to-be-written data block based on the data block name of the to-be-written data block and a data server list obtained from the metadata server. The to-be-written data block allocated to the local data server is sent to the local data server, or the to-be-written data block allocated to other data servers is sent to the other data servers through the local data server. The data servers share network communication, and there is no need to maintain an independent link for each data server. The data block of the to-be-written data is only sent to the allocated data server, rather than all data servers, so that concurrent duplicate data streams are eliminated, bandwidth overload is avoided, the risk of network congestion is reduced, and the utilization rate of the storage resources of the data servers is improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] The above and other features, advantages, and aspects of the present disclosure will become more apparent by describing in detail the following specific embodiments thereof with reference to the attached drawings. Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. It should be understood that the drawings are schematic and elements are not necessarily to scale.

[0047] Figure 1 A flowchart of a data writing method provided by an embodiment of the application;

[0048] Figure 2 A schematic diagram of a file and data shards provided by an embodiment of the application;

[0049] Figure 3 A schematic diagram of a file, a shard, and a data block provided by an embodiment of the application;

[0050] Figure 4 A structural block diagram of a data server and a metadata server provided by an embodiment of the application;

[0051] Figure 5A flowchart of another data writing method provided by an embodiment of the present application is shown in FIG. 2.

[0052] Figure 6 A flowchart of another data writing method provided by an embodiment of the present application is shown in FIG. 2.

[0053] Figure 7 A hardware structure diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 3. DETAILED DESCRIPTION

[0054] The embodiments of the present application are described below in conjunction with the accompanying drawings. The terms used in the embodiments section of the present application are only used to explain the embodiments of the present application, and are not intended to limit the present application.

[0055] The embodiments of the present application are described below in conjunction with the accompanying drawings. The terms used in the embodiments section of the present application are only used to explain the embodiments of the present application, and are not intended to limit the present application.

[0056] The terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and are not necessarily intended to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged, and this is only a way of distinguishing the objects with the same attributes in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or devices containing a series of units do not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or devices.

[0057] In order to reduce the risk of network congestion and improve the utilization rate of storage resources of data servers, the present application provides a data writing method, which is applied to any data server, and each data server is in communication connection with a metadata server. The data writing method provided by the present application is described in further detail below in conjunction with the accompanying drawings and the specific embodiments.

[0058] It should be noted that the method is run on any data server (physical or logical service actually storing data content, with data read-write and local computing capability) in a distributed storage system, wherein multiple data servers can be directly connected to each other through a TCP network (a full-duplex communication protocol that guarantees transmission reliability and sequence), and each data server independently establishes a dedicated communication connection channel with the central coordination node, i.e., the metadata server, thereby constructing a double network topology: forming an efficient data exchange transmission plane between data servers, and constructing a precise instruction interaction control plane between data servers and metadata services.

[0059] Among them, the data server is the physical or virtualized basic unit of the distributed file system, each data server is equipped with independent local disk resources, and realizes direct communication with other data servers through the TCP network to form an efficient data block transmission channel. These nodes jointly bear the file content storage and cross-node exchange tasks, and are the entity carrier of the system mass data landing. The client module is the core software component of the data service node, which runs in the form of a process and provides a standard file system interface based on the FUSE (Filesystem in Userspace) framework. The local disk undertakes the final data persistence responsibility.

[0060] The metadata server is the core coordination hub of the distributed file system, which supports independent node deployment and can be mixed with data nodes in the same physical machine. The data server stores actual data, and the metadata server manages the meta information (such as location) describing these data. The architecture in this embodiment contains multiple key modules working together: interface module, index management module and node management module. Specifically, the interface module serves as a unified entrance to support communication between data servers and metadata servers, providing standardized remote service access and realizing protocol standardization and cross-platform interaction capability. The index module builds the file index of the entire system, and the metadata of the valid index node in the file index further contains multiple shards, each shard size is not fixed, which represents the data of one-time continuous writing. The node management module builds and maintains the data server list of multiple data server information of the entire system. When the data server starts, the node ID and node IP address of the data server are stored in the node management module through the interface module, and the module synchronously implements heartbeat monitoring to automatically exclude faulty data servers and ensure service reliability.

[0061] Since the ranges between the shards of the write data can overlap, the write data can have the case of repeated writing with coverage, resulting in storage redundancy, so a GC module can also be added. The GC module periodically scans the shards of the write data. When detecting that there are overlapping shards, the old shard data that is completely covered is automatically cleared and the partially overlapping area is merged, the space recycling mechanism effectively compresses the metadata index size, reduces the read operation complexity from O(N) to O(1), and at the same time improves the storage space utilization.

[0062] Please refer to the accompanying Figure 1 , Figure 1 A flowchart of a data writing method provided by an embodiment of the present application is shown. The method can include the following steps:

[0063] Step S101: receiving a data writing request, the data writing request carrying a file name of a file in which to-be-written data is located and a logical position of the to-be-written data in the file in which the to-be-written data is located.

[0064] It should be noted that the data writing request refers to an operation instruction initiated by an application program to a data server through a client module, requiring to write specific data to a data storage device (such as a disk) of the data server. The to-be-written data refers to binary content required to be written by the application program. The file name of the file in which the to-be-written data is located refers to a logical identifier of a storage unit in which the to-be-written data is located. The logical position of the to-be-written data in the file in which the to-be-written data is located refers to a logical position of the to-be-written data in the file in which the to-be-written data is located, usually including a start offset of the to-be-written data relative to the file in which the to-be-written data is located and an end offset of the to-be-written data relative to the file in which the to-be-written data is located.

[0065] In the present application, the data server receives a data writing request initiated by an application program through a client module, and the data writing request only needs to specify the logical information of the file in which the to-be-written data is located, that is, the file name (such as testfile1) and the logical position (such as 1024 bytes to 10485760 bytes), rather than the physical data itself. This design only needs to lock the operation object based on lightweight data.

[0066] Further, before receiving the data writing request, the method can further include the following steps: receiving a file creation request of a target file, the file creation request carrying a file creation name and a file creation path; sending the file creation request to the metadata server, so that the metadata server creates the target file based on a target creation path determined by parsing the file creation path, and the name of the target file is the file creation name.

[0067] It should be noted that a target file creation request refers to an operation instruction initiated by the application through the client module to the data server, requesting the creation of a new target file under a specified storage path. The file creation name is the logical identifier of the target file. The file creation path is the logical path of the target file, consisting of multiple levels of directories.

[0068] In this application, the data server receives a file creation request for a target file initiated by the application through the client module. The file creation request includes the file creation name (e.g., access.log) and the file creation path (e.g., / data / logs / ). The data server forwards the file creation request to the interface module of the metadata server through the client module, triggering the following operations: The metadata server sends the request to the index management module through the interface module to parse the file creation path, querying the path nodes level by level. First, it finds the index node of the subdirectory 'data' in the root directory, then finds the index node of the subdirectory 'logs' in the 'data' directory. The index node of the path endpoint 'logs' directory no longer contains subdirectories, i.e., leaf index nodes. Then, the leaf index nodes can be used as the logical parent container of the file to be created, inheriting its permissions, ownership, and other attributes, thereby determining the target creation path and creating the target file; the file creation name can be used as the name of the target file (e.g., access.log). Alternatively, a globally unique index node ID can be assigned to the newly created index node. The index node ID of the new file can be derived from the leaf index node ID (e.g., hash overlay) to ensure hierarchical association.

[0069] Step S102: Send a query request to the metadata server. The query request carries the file name of the file where the data to be written is located.

[0070] In this application, a query request is a standardized network command issued by the client module of the data server to the interface module of the metadata server, used to proactively initiate a targeted query to the metadata server. The core content of this query request carries the filename of the file containing the data to be written, submitted by the user-side device. Its fundamental purpose is to resolve the physical storage information corresponding to that filename through the metadata service.

[0071] Step S103: Receive the data fragment ID assigned to the data to be written by the metadata server. The data fragment ID of the data to be written is the data fragment ID obtained by the metadata server based on the file name of the file where the data to be written is located.

[0072] In this application, the index management module of the metadata server receives the file name of the file containing the data to be written through the interface module, and queries the preset file index based on the file name of the file containing the data to be written to obtain the data shard ID (such as 1010) assigned to the data to be written. Then, the index management module of the metadata server sends the data shard ID assigned to the data to be written obtained through the interface module to the client module of the data server.

[0073] It should be noted that the default file index is a global key-value database maintained internally by the index management module in the metadata server. This database includes at least the filename of the file containing the data to be written and the data shard ID of the data to be written. For a valid inode in the file index, its metadata further contains multiple shards of the data to be written. Each shard represents a segment of data written consecutively. The size of each shard is not fixed, and each shard has a globally unique, auto-incrementing data shard ID. The data shard ID is the unique identifier of the shard. For a clearer understanding, please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram of file and data fragmentation provided in an embodiment of this application. In the diagram, file represents a file, and slice1, slice2, and slice3 represent data fragment IDs.

[0074] Step S104: Divide the received data to be written into blocks to obtain at least one data block to be written, and set a data block name for the data block to be written based on the data fragment ID of the data to be written and the logical position of the data block to be written in the data to be written.

[0075] It should be noted that the client module in the data server receives the data to be written sent by the application. The data block to be written is the smallest physical storage entity in the data storage device of the data server. It typically uses a fixed size to adapt to the characteristics of the storage medium, and each data block to be written is physically independent. The logical position of the data block to be written within the data to be written refers to its logical location within the data, which usually includes the start offset and end offset of the data block relative to the data to be written.

[0076] In this application, when the client module in the data server receives the data to be written and the data fragment ID of the data to be written sent by the application, it first divides the continuous data stream to be written into multiple physical storage units according to a preset fixed size (e.g., 4MB). Each unit is called a data block to be written. This standardized block design is deeply adapted to the characteristics of the storage medium. For example, the 4MB block size perfectly matches the page alignment requirements of SSDs, which can significantly improve writing efficiency and extend device life. During the block division process, it is necessary to handle the tail data scenario. When the remaining data is less than the standard size, it is allowed to generate a tail block smaller than the standard size to ensure data integrity. For a better understanding, please refer to the following for details. Figure 3 , Figure 3 This diagram illustrates a file, slice, and data block according to an embodiment of this application. In the diagram, "file" represents a file, "slice" represents a slice, and "block" represents a data block. For each data block to be written, a three-segment naming structure is used: <Data Slice ID of the data to be written>_<Start offset of the data block to be written relative to the data to be written>_<End offset of the data block to be written relative to the data to be written>, resulting in the data block name (e.g., 1010_0_4194303, 1010_4194304_8388607, and 1010_8388608_10484736). Essentially, this naming mechanism uses the self-describing data slice ID of the data to be written as a logical anchor point. The start and end offsets form a closed interval to precisely mark the position of the data block to be written relative to the data to be written, enabling the name itself to simultaneously carry the triple functions of physical location, logical attribution, and integrity verification.

[0077] Step S105: Based on the list of data servers obtained from the metadata server, obtain the number of nodes of the data servers, and allocate at least one data server for the data block to be written based on the data block name and the number of nodes of the data servers.

[0078] It should be noted that the data server list is a real-time online collection of storage node addresses maintained by the node management module of the metadata server. The data server list includes node information for multiple data servers, specifically at least the node ID and IP address of the data server. The list is automatically updated when a data server fails or is expanded, ensuring that computational allocation is always based on the latest cluster status.

[0079] In this application, the hash value of the data block to be written can first be calculated using a hash algorithm based on the data block name. Then, at least one data server can be allocated to the data block to be written based on the hash value of the data block to be written and the number of nodes of the data servers in the data server list obtained from the metadata server.

[0080] Specifically, the client module of the data server uses a preset hash algorithm (such as MD5 or SHA-256) to mathematically transform the name of the data block to be written, generating a fixed-length digital fingerprint. Taking the data block name 1010_4194304_8388607 as an example, inputting this string into the MD5 algorithm will output a 128-bit hash value d41d8cd98f00b204e9800998ecf8427e. The essence of this transformation is to compress the variable-length name into a fixed-length digest, ensuring that different block names obtain a uniformly distributed hash value, laying a mathematical foundation for subsequent load balancing.

[0081] The system then retrieves a list of currently active data servers from the metadata server, including the node IDs (e.g., 0, 1, and 2) and node IP addresses (e.g., 192.168.1.101, 192.168.1.102, and 192.168.1.103) of all available data servers. A modulo operation is performed based on the hash value of the data block to be written and the number of nodes on the data server (e.g., 3): Node index = Hash value of the data block to be written % Number of nodes on the data server. For example, converting a 128-bit MD5 hash value to an integer and taking the modulo of 3 yields 2, thus selecting the data server with node ID 2. This deterministic mapping ensures that the same block name always points to the same data server, while the avalanche effect of the hash function ensures that data blocks are evenly distributed across all data servers.

[0082] Step S106: Send the data block to be written allocated to the local data server to the local data server, or send the data block to be written allocated to other data servers to other data servers through the local data server. Other data servers are data servers other than the local data server.

[0083] It should be noted that in a distributed storage architecture, data servers assume different roles based on their position in write transactions, forming an efficient collaborative network. Data servers include local data servers and other data servers. The local data server acts as a transaction coordinating node, directly receiving write requests from the client modules of other data servers; the other data servers' client modules receive data blocks forwarded from the local data server's client modules, acting as storage execution nodes.

[0084] Once the data server allocates data blocks for writing, for data blocks allocated to the local data server (i.e., the server currently performing the write operation), there's no need for network forwarding. Instead, the local client module directly transmits the data block to the local storage service module and writes it to the local disk with the corresponding data block name. For data blocks allocated to other data servers, the local data server's client module first uses a pre-established TCP network channel to forward the data blocks allocated to other data servers to the client modules of those other data servers. Taking data block 1010_4194304_8388607 as an example, after hash modulo calculation to locate the data server 192.168.1.103 with node ID 2, the local data server's client module encapsulates the data block and its name into a transmission unit and sends it to the client modules of other data servers via a multiplexed TCP connection. The client modules of other data servers employ a pre-write verification mechanism, which checks the logical integrity of the data block name and the hash value matching of the data block to be written. Once verified, the data block is stored on the disk of the other data server using the data block name as the filename (e.g., local_dir / 1010_4194304_8388607.blk). This process is completely decentralized; each data block to be written is only transmitted to the actual storage data server.

[0085] After sending the data blocks to be written allocated to the local data server to the local data server, and sending the data blocks to be written allocated to other data servers to the other data servers through the local data server, the following steps may also be included: First, a query request may be sent to the metadata server, the query request carrying the file name of the file where the data to be written is located; then, the inode ID of the file where the data to be written is located may be received from the metadata server, the inode ID of the file where the data to be written is located is obtained by querying a preset file index based on the file name of the file where the data to be written is located, the preset file index includes the file names and inode IDs of multiple files; then, the data name of the data to be written may be set based on the logical position of the data to be written in the file where the data to be written is located and the data fragment ID of the data to be written; finally, the inode ID of the file where the data to be written is located and the data name of the data to be written may be sent to the metadata server, so that the metadata server updates the preset file index based on the inode ID of the file where the data to be written is located and the data name of the data to be written.

[0086] It should be noted that a query request is a standardized network command issued by the client module of the data server to the interface module of the metadata server, used to proactively initiate a targeted query to the metadata server. The core content of this query request carries the filename of the file containing the data to be written, submitted by the user device. Its fundamental purpose is to parse the physical storage information corresponding to that filename through the metadata service.

[0087] The default file index is a global key-value database maintained internally by the index management module in the metadata server. It includes the filename of the file containing the data to be written, the inode ID of that file, and the mapping between the logical location of the data to be written within that file. Each file containing the data to be written has an inode ID, which is a unique identifier for the inode of the file containing the data to be written.

[0088] Specifically, the metadata server's index management module receives the file name of the file containing the data to be written through the interface module, and queries the preset file index based on the file name to obtain the inode ID of the stored file. Then, the metadata server's index management module sends the obtained inode ID of the file to be written to the data server's client module through the interface module.

[0089] The name of the data to be written is composed of the data fragment ID of the data to be written and the logical position of the data to be written in the file where the data to be written is located. It is a logically unique identifier for the data to be written. Specifically, the client module of the data server adopts a three-segment naming structure for the data to be written: <data fragment ID of the data to be written>_<start offset of the data to be written relative to the file where the data to be written is located>_<end offset of the data to be written relative to the file where the data to be written is located>, to obtain the name of the data to be written (e.g., 1010_1024_10485760). This naming mechanism essentially uses the data fragment ID of the data to be written as a logical anchor point, and the start offset and end offset form a closed interval to accurately mark the position of the data to be written relative to the file where the data to be written is located, so that the name itself simultaneously carries the triple functions of physical location, logical ownership, and integrity verification.

[0090] Once all data blocks have been transmitted, the data server sends the inode ID (e.g., 1234) of the file containing the data to be written, along with the data name (e.g., 1010_1024_10485760), to the metadata server. Upon receiving this information, the metadata server triggers index tree reconstruction. This involves locating the metadata structure corresponding to the inode ID of the file containing the data to be written within the predefined file index, and appending the data name as a new entry to the data's fragment entry (e.g., inserting 1010_1024_10485760 at the end of the fragment entry corresponding to the data 1234).

[0091] For better understanding, please refer to the following details. Figure 4 , Figure 4 This is a schematic diagram of the structure of a data server and a metadata server provided in an embodiment of this application. In the diagram, metadata service represents a metadata server, data service represents a data service, inode represents an index node ID, name represents a file name, slice represents a segment, and key represents a data block name.

[0092] In summary, this application provides a data writing method applicable to any data server. The data servers communicate with each other, and each data server also communicates with a metadata server. The method receives the filename of the file containing the data to be written, and the logical position of the data within that file. It then sends the filename to the metadata server and receives a data fragment ID assigned by the metadata server based on that filename. The introduction of the data fragment ID allows data to be decoupled into manageable fragments, reducing the initial data transfer volume and the size of the transmission unit. The received data to be written is divided into blocks to obtain at least one data block. A data block name is set for each data block based on its data fragment ID and logical position within the data. At least one data server is assigned to each data block based on its data block name and a list of data servers obtained from the metadata server. Data blocks assigned to a local data server are sent to the local data server, or data blocks assigned to other data servers are sent to other data servers via the local data server. Data servers share network communication, eliminating the need to maintain independent links for each data server. Data blocks to be written are only sent to the assigned data server, not all data servers, eliminating concurrent duplicate data streams, avoiding bandwidth overload, reducing the risk of network congestion, and each data server only needs to store the data of the required data blocks to be written, improving the utilization rate of data server storage resources.

[0093] Based on the embodiments disclosed in this application above, another embodiment of this application provides a detailed description of the specific implementation method for data reading. Please refer to the appendix. Figure 5 , Figure 5 This is a flowchart illustrating another data writing method provided in an embodiment of this application.

[0094] As one possible implementation, after sending the inode ID of the file containing the data to be written and the data name of the data to be written to the metadata server, the method may include the following steps:

[0095] Step S201: Send a data read request to the metadata server. The data read request carries the file name of the file containing the data to be read, as well as the logical position of the data to be read in the file containing the data to be read.

[0096] A data read request is an operation instruction initiated by an application through its client module to a data server, requesting the reading of specific data from the data server's data storage device (such as a disk). The data to be read refers to the binary content that the application requests to read. The filename of the file containing the data to be read is the logical identifier of the storage unit where the data is located. The logical position of the data to be read within the file refers to its logical location within the file, typically including the start and end offsets of the data relative to the file's contents.

[0097] In this application, the data server receives data read requests initiated by the application through the client module. The data read request only needs to specify the logical information of the file where the data to be read is located, namely the file name (e.g., testfile2) and logical location (e.g., 1024_4096), rather than the physical data itself. This design can lock the operation object based on lightweight data.

[0098] Step S202: Receive the data fragment ID of the stored data and the logical position of the stored data in the file where the data to be read is located, returned by the metadata server based on the file name of the file where the data to be read is located and the logical position of the data to be read in the file where the data to be read is located.

[0099] In this application, when the client module of the data server initiates a read request to the interface module of the metadata server, the index management module of the metadata server receives the file name (e.g., testfile2) and the logical position (e.g., bytes 1024 to 4096) of the file containing the data to be read through the interface module. First, it scans the file index using a range matching algorithm to identify the logical position of the already stored data covering the data to be read within the file containing the data to be read, quickly locating the already stored data containing the data to be read. Taking testfile2 as an example, the logical position (bytes 1024 to 4096) of the data to be read within the logical position (bytes 1024-10485760) of the already stored data within the file containing the data to be read, thus obtaining the data fragment ID of the already stored data as 1010. The index management module of the metadata server then returns the data fragment ID of the already stored data containing the data to be read and the logical position of the already stored data within the file containing the data to be read to the data server through the interface module, completing the first conversion from logical address to physical storage.

[0100] Step S203: Based on the logical position of the data to be read in the file where the data to be read is located, the data fragment ID of the stored data where the data to be read is located, and the logical position of the stored data in the file where the data to be read is located, set the data block name for the data block to be read.

[0101] In this application, the logical position of the data block to be read within the data to be read can first be calculated based on the logical position of the data to be read within the file containing the data to be read and the logical position of the stored data within the file containing the data to be read. Then, the data block name can be set for the data block to be read based on the data fragment ID of the stored data containing the data to be read and the logical position of the data block to be read within the data to be read.

[0102] Specifically, the client module of the data server first calculates the starting and ending offsets of the data block to be read relative to the data to be read, based on the starting and ending offsets of the data to be read relative to the file containing the data to be read, and the starting and ending offsets of the stored data relative to the file containing the data to be read. This transformation process is essentially a coordinate system translation, mapping the global coordinates of the file to the local coordinates of the fragment. The specific formulas are as follows: Starting offset of the data block to be read relative to the data to be read = Starting offset of the file containing the data to be read - Starting offset of the stored data relative to the file containing the data to be read; Ending offset of the data block to be read relative to the data to be read = Ending offset of the file containing the data to be read - Starting offset of the stored data relative to the file containing the data to be read. For example, when the logical position of the data to be read in the file containing the data to be read is between bytes 1024 and 4096, and the logical position of the stored data in the file containing the data to be read is between bytes 1024 and 10485760, the logical position of the data block to be read in the data to be read is calculated to be between bytes 0 and 3072. Since the standard size of the data block to be read is 4MB, and the logical positions of the data block to be read from byte 0 to byte 3072 in the data to be read are completely contained within the first data block from byte 0 to byte 4194303, and the data fragment ID of the stored data where the data to be read is located is known to be 1010, the data block name of the data block to be read is 1010_0_4194303.

[0103] Step S204: Based on the data block name of the data block to be read and the list of data servers obtained from the metadata server, determine the data server where the data block to be read is located.

[0104] In this application, the client module of the data server can determine the data server where the data block to be read is located as 192.168.1.101 based on the data block name 1010_0_4194303 of the data block to be read set and the data server list 192.168.1.101, 192.168.1.102 and 192.168.1.103 obtained by the metadata server through hash modulo operation.

[0105] Step S205: Read the data to be read from the local data server where the data block to be read is located, or read the data to be read from another data server where the data block to be read is located through the local data server.

[0106] In this application, if the data block to be read is allocated to the local data server, i.e., the server currently processing the read request itself, no network communication needs to be initiated. The local data server directly locates the corresponding file in its local storage directory based on the data block name, and precisely reads the data content at the specified logical position through the file system's range read interface, and then directly returns the data to the upper-layer application; the entire process is completed locally. If the data block to be read is located on another data server, the local data server sends a precise read request to the other data server 192.168.1.101 through a pre-established TCP connection. The request packet explicitly specifies that data from byte 0 to byte 3072 of data block 1010_0_4194303 needs to be obtained. After receiving the request, the other data server 192.168.1.101 directly locates the file named 1010_0_4194303.blk in the local storage directory local_dir, precisely extracts the first 3072 bytes of content through the file system's range read interface (such as Linux's pread), and returns the data through the same TCP connection. After receiving the data, the local data server immediately forwards it to the upper-layer application and releases the relevant network resources, completing a full distributed and precise read operation.

[0107] In summary, this application provides a data reading method. First, a data reading request is sent to a metadata server, carrying the filename of the file containing the data to be read, and the logical position of the data within that file. The metadata server then returns the data fragment ID of the stored data containing the data to be read, along with its corresponding logical position, based on the filename and logical position of the stored data. In response, the metadata server returns the data fragment ID and its corresponding logical position based on the filename and logical position. This ensures that the returned information accurately matches the user's needs, eliminating the need to query the entire file's metadata, significantly reducing response time and improving query efficiency. Next, based on the logical position of the data to be read, the data fragment ID, and the logical position of the stored data, a data block name is set for the data block. This step achieves a unique identifier and logical mapping for the data block, effectively avoiding data location errors or redundant readings. Then, by combining the data block name with the data server list obtained from the metadata server, the specific data server containing the data block is quickly determined. This distributed location mechanism reduces continuous intervention from the metadata server and achieves load balancing. This not only improves the overall scalability of the system, especially in large-scale distributed environments where it can support more concurrent requests, but also ensures the reliability and low latency of data transmission by directly reading data through a peer-to-peer network.

[0108] To reduce the risk of network congestion, this application provides a data writing method. This method is applied to a metadata server, which is communicatively connected to multiple data servers, and the data servers are communicatively connected to each other. The data writing method provided by this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0109] Please see the appendix Figure 6 , Figure 6 This is a flowchart illustrating another data writing method provided in an embodiment of this application. The method may include the following steps:

[0110] Step S301: Receive the query request sent by the data server after receiving the data write request. The data write request carries the file name of the file where the data to be written is located, and the logical position of the data to be written in the file where the data to be written is located. The query request carries the file name of the file where the data to be written is located.

[0111] It should be noted that a data write request refers to an operation instruction initiated by an application through its client module to the data server, requesting that specific data be written to the data server's data storage device (such as a disk). The data to be written refers to the binary content that the application requests to be written. The filename of the file containing the data to be written is the logical identifier of the storage unit where the data is located. The logical position of the data to be written within the file refers to its logical location within the file, typically including the start and end offsets of the data relative to the file's contents.

[0112] In this application, the data server receives data write requests initiated by the application through the client module. The data write request only needs to specify the logical information of the file containing the data to be written, i.e., the file name (e.g., testfile1) and logical location (e.g., 1024_10485760), rather than the physical data itself. This design allows for the identification of the operation object based solely on lightweight data. A query request is a standardized network command issued by the data server's client module to the metadata server's interface module, used to proactively initiate a targeted query to the metadata server. The core content of this query request carries the file name of the file containing the data to be written submitted by the user-side device. Its fundamental purpose is to parse the physical storage information corresponding to that file name through the metadata service.

[0113] Step S302: Assign a data fragment ID based on the filename of the file containing the data to be written.

[0114] In this application, a new data segment ID is assigned to the data to be written based on the data segment ID of the file corresponding to the file name of the file containing the data to be written, which is in the preset file index; a new data segment ID is assigned to the data to be written based on the data segment ID of the file corresponding to the file name of the file containing the data to be written.

[0115] It should be noted that the default file index is a global key-value database maintained internally by the index management module in the metadata server. It includes at least the filename of the file containing the data to be written, the inode ID of that file, the data fragment ID of the data to be written, and the mapping relationship between the logical location of the data to be written within that file. Each file containing the data to be written has an inode ID, which is a unique identifier for the inode of that file. For a valid inode in the file index, its metadata further contains multiple fragments of the data to be written, representing a continuous data segment being written. The size of each fragment is not fixed, and each fragment has a globally unique, auto-incrementing data fragment ID, which is the unique identifier for the fragment of the data to be written.

[0116] Specifically, the process begins by querying the preset file index based on the filename of the file containing the data to be written, obtaining the inode ID and the data shard ID of the stored data (e.g., 1010) corresponding to the filename of the file containing the data to be written. Then, the inode ID corresponding to the filename of the file containing the data to be written is determined as the inode ID of the file containing the data to be written, and a new data shard ID is assigned to the data to be written based on the data shard ID of the stored data. The metadata server's index management module receives the filename of the file containing the data to be written through the interface module, queries the preset file index based on the filename of the file containing the data to be written to obtain the inode ID and the data shard ID of the data to be written (e.g., 1010). The metadata server's index management module then determines the inode ID corresponding to the filename of the file containing the data to be written as the inode ID of the file containing the data to be written, and assigns a new data shard ID to the data to be written by auto-incrementing based on the data shard ID of the stored data.

[0117] Step S303: Send the data fragment ID of the data to be written to the data server so that the data server can divide the received data to be written into blocks to obtain at least one data block to be written.

[0118] In this application, the inode ID of the file to be written and the data fragment ID assigned to the data to be written can be sent to the client module of the data server through the interface module. The client module of the data server adopts a three-segment naming structure for the data to be written: <data fragment ID of the data to be written>_<start offset of the data to be written relative to the file where the data to be written is located>_<end offset of the data to be written relative to the file where the data to be written is located>, to obtain the data name of the data to be written (e.g., 1010_1024_10485760). This naming mechanism essentially uses the data fragment ID of the data to be written as a logical anchor point, and the start offset and end offset form a closed interval to accurately mark the position of the data to be written relative to the file where the data to be written is located, so that the name itself simultaneously carries the triple functions of physical location, logical ownership, and integrity verification. In this application, when the client module in the data server receives the data to be written and the data fragment ID of the data to be written sent by the application, it first divides the continuous data stream to be written into multiple physical storage units according to a preset fixed size (such as 4MB). Each unit is called a data block to be written. This standardized block design is deeply adapted to the characteristics of the storage medium. For example, the 4MB block size perfectly matches the page alignment requirements of SSD, which can significantly improve writing efficiency and extend device life. During the block division process, it is necessary to handle the tail data scenario. When the remaining data is less than the standard size, it is allowed to generate a tail block smaller than the standard size to ensure data integrity.

[0119] It should be noted that the name of the data to be written is composed of the data fragment ID of the data to be written and the logical position of the data to be written in the file where the data to be written is located, and is a logically unique identifier for the data to be written. The client module in the data server receives the data to be written sent by the application. The data block to be written is the smallest physical storage entity of the data storage device in the data server. It usually adopts a fixed size to adapt to the characteristics of the storage medium, and each data block to be written has physical independence.

[0120] Step S304: Send the data server list to the data server so that the data server allocates at least one data server for the data block to be written based on the name of the data block to be written and the number of data server nodes. The data server sends the data block to be written allocated to the local data server to the local data server, or sends the data block to be written allocated to other data servers to other data servers through the local data server. The name of the data block to be written is set by the data server based on the data fragment ID of the data to be written and the logical position of the data block to be written in the data to be written. The number of data server nodes is obtained based on the data server list. Other data servers are data servers other than the local data server.

[0121] It should be noted that the logical position of the data block to be written within the data to be written refers to its logical location within the data, typically including the start and end offsets of the data block relative to the data to be written. The data server list is a set of real-time online storage node addresses maintained by the node management module of the metadata server. Each element in the data server list must include at least the node ID and IP address of the data server. The list is automatically updated when a data server fails or is expanded, ensuring that computational allocation is always based on the latest cluster state.

[0122] The client module of the data server can first calculate the hash value of the data block to be written using a hash algorithm based on the data block name. Then, based on the hash value of the data block to be written and the number of data servers in the data server list obtained from the metadata server, it can allocate at least one data server for the data block to be written.

[0123] Data servers assume different roles based on their position in write transactions, forming an efficient collaborative network. Data servers include local data servers and other data servers. The local data server acts as a transaction coordinating node, directly receiving write requests from the client modules of other data servers; the client modules of other data servers act as storage execution nodes, receiving data blocks forwarded from the client modules of the local data server.

[0124] In this application, a three-segment naming structure is adopted for each data block to be written: <Data Fragment ID of the data to be written>_<Start offset of the data block to be written relative to the data to be written>_<End offset of the data block to be written relative to the data to be written>, to obtain the data block name of the data block to be written (such as 1010_0_4194303, 1010_4194304_8388607, and 1010_838860_10485760). This naming mechanism essentially uses the data fragment ID of the data to be written as a self-describing storage point as a logical anchor point. The start and end offsets form a closed interval to precisely mark the position of the data block to be written relative to the data to be written, enabling the name itself to simultaneously carry the triple functions of physical location, logical attribution, and integrity verification. The client module of the data server uses a preset hash algorithm (such as MD5, SHA-256) to mathematically transform the data block name of the data block to be written, generating a fixed-length digital fingerprint. Taking the data block name 1010_4194304_8388607 as an example, inputting this string into the MD5 algorithm will output a 128-bit hash value d41d8cd98f00b204e9800998ecf8427e. The essence of this transformation is to compress a variable-length name into a fixed-length digest, ensuring that different block names obtain evenly distributed hash values, laying the mathematical foundation for subsequent load balancing. Then, a list of currently active data servers is obtained from the metadata server, containing the node IDs (e.g., 0, 1, and 2) and node IP addresses (e.g., 192.168.1.101, 192.168.1.102, and 192.168.1.103) of all available data servers. The modulo operation is performed based on the hash value of the data block to be written and the number of data servers (e.g., 3): Node index = Hash value of the data block to be written % Number of data servers. For example, converting a 128-bit MD5 hash value to an integer and taking the modulo of 3 yields 2, so the data server with node ID 2 is selected. This deterministic mapping ensures that the same block name always points to the same data server, while the avalanche effect of the hash function ensures that data blocks are evenly distributed across all data servers.

[0125] After the data server allocates data blocks to be written, for data blocks allocated to other data servers, the local data server's client module first transmits the allocated data blocks to the client modules of those other data servers via a pre-established TCP network channel. Taking data block 1010_4194304_8388607 as an example, after hash modulo calculation to locate data server 192.168.1.103 with node ID 2, the local data server's client module encapsulates the data block and its name into a transmission unit and sends it to the client modules of other data servers via a multiplexed TCP connection. The client modules of the other data servers use a pre-write verification mechanism, checking the logical integrity of the data block name and the hash value matching. After confirming that everything is correct, the file is stored on the other data server's disk using the data block name as the filename (e.g., local_dir / 1010_4194304_8388607.blk). This process is completely decentralized; each block of data to be written is only transmitted to the actual storage data server.

[0126] Furthermore, once it is confirmed that all data blocks have been transmitted, the data server sends the inode ID (e.g., 1234) of the file containing the data to be written and the data name (e.g., 1010_1024_10485760) of the data to be written to the metadata server. Upon receiving this information, the metadata server triggers index tree reconstruction, which involves locating the metadata structure corresponding to the inode ID of the file containing the data to be written in the preset file index, and appending the data name of the data to be written as a new entry to the data fragment entry (e.g., inserting 1010_1024_10485760 at the end of the fragment entry corresponding to the data 1234).

[0127] In summary, this application provides a data writing method. This method is applied to a metadata server, which is communicatively connected to multiple data servers, and the data servers are also communicatively connected to each other. The method receives a query request sent by a data server after receiving a data write request. The data write request carries the filename of the file containing the data to be written, and the logical position of the data to be written within that file. The query request carries the filename of the file containing the data to be written. Based on the filename of the file containing the data to be written, the method queries a preset file index to obtain the inode ID of the file containing the data to be written, and assigns a data fragment ID to the data to be written. The method then sends the inode ID of the file containing the data to be written and the data fragment ID of the data to be written to the data servers, so that the data servers can use the logical position of the data to be written within the file to determine its position. The process involves setting the data name for the data to be written using the data fragment ID and dividing the received data into blocks to obtain at least one data block. A list of data servers is sent to the data servers, enabling them to allocate at least one data server for the data block based on its name and the number of nodes. The data servers can also send data blocks allocated to their local servers to the local data server, or send data blocks allocated to other data servers to other data servers via the local data server. The name of the data block is set by the data server based on the data fragment ID and its logical position within the data to be written. The number of data server nodes is obtained from the data server list. The introduction of the data fragment ID allows data to be decoupled into manageable fragments, reducing initial data transfer volume and transmission unit size. Data servers share network communication, eliminating the need to maintain independent links for each data server. Data blocks are only sent to their assigned data servers, not all data servers, eliminating concurrent duplicate data streams, avoiding bandwidth overload, and reducing network congestion risk. Furthermore, by centrally managing file mappings using a metadata server, the need for each data server to maintain independent data copies is avoided. This design completely delegates subsequent data block routing decisions to the data servers, eliminating the need for the metadata server to initiate additional queries. Consequently, the metadata server transforms into a lightweight write coordinator, significantly reducing its own access pressure.

[0128] This application also provides an electronic device in its embodiments. (See reference...) Figure 7 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0129] like Figure 7 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. When the electronic device is powered on, the RAM 703 also stores various programs and data required for the operation of the electronic device. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0130] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, memory cards, hard drives, etc.; and communication devices 709. Communication device 709 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0131] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the data writing methods provided in this application.

[0132] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the data writing methods provided in this application.

[0133] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0135] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0136] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data writing method, characterized in that, The method is applied to any data server, wherein the data servers communicate with each other, and each data server communicates with a metadata server. The method includes: Receive a data write request, the data write request carrying the file name of the file where the data to be written is located, and the logical position of the data to be written in the file where the data to be written is located; A query request is sent to the metadata server, the query request carrying the file name of the file containing the data to be written; Receive the data fragment ID allocated for the data to be written returned by the metadata server. The data fragment ID of the data to be written is the data fragment ID obtained by the metadata server based on the file name of the file where the data to be written is located. The received data to be written is divided into blocks to obtain at least one data block to be written, and a data block name is set for the data block to be written based on the data fragment ID of the data to be written and the logical position of the data block to be written in the data to be written. Based on the list of data servers obtained from the metadata server, the number of nodes of the data servers is obtained, and at least one of the data servers is allocated for the data block to be written based on the data block name of the data block to be written and the number of nodes of the data servers. The data block to be written, which is allocated to the local data server, is sent to the local data server, or the data block to be written, which is allocated to other data servers, is sent to the other data servers through the local data server, wherein the other data servers are data servers other than the local data server.

2. The data writing method according to claim 1, characterized in that, After sending the data block to be written allocated to the local data server to the local data server, or sending the data block to be written allocated to other data servers through the local data server to the other data servers, the method further includes: A query request is sent to the metadata server, the query request carrying the file name of the file containing the data to be written; The system receives the inode ID of the file containing the data to be written, returned by the metadata server. The inode ID of the file containing the data to be written is obtained by querying a preset file index based on the file name of the file containing the data to be written. The preset file index includes the file names and inode IDs of multiple files. Based on the logical position of the data to be written in the file where the data to be written is located and the data fragment ID of the data to be written, a data name is set for the data to be written; The inode ID of the file containing the data to be written and the data name of the data to be written are sent to the metadata server, so that the metadata server updates the preset file index based on the inode ID of the file containing the data to be written and the data name of the data to be written.

3. The data writing method according to claim 2, characterized in that, Prior to receiving the data write request, the method further includes: Receive a file creation request for the target file, the file creation request carrying the file creation name and file creation path; The file creation request is sent to the metadata server so that the metadata server creates the target file based on the target creation path determined by parsing the file creation path, and the name of the target file is the file creation name.

4. The data writing method according to claim 2, characterized in that, After sending the inode ID of the file containing the data to be written and the data name of the data to be written to the metadata server, the method further includes: Send a data read request to the metadata server. The data read request carries the file name of the file containing the data to be read, and the logical position of the data to be read in the file containing the data to be read. The metadata server receives the data fragment ID of the stored data and the logical position of the stored data in the file where the data to be read is located, based on the file name of the file where the data to be read is located and the logical position of the data to be read in the file where the data to be read is located. Based on the logical position of the data to be read in the file where the data to be read is located, the data fragment ID of the stored data where the data to be read is located, and the logical position of the stored data in the file where the data to be read is located, a data block name is set for the data block to be read; Based on the data block name of the data block to be read and the list of data servers obtained from the metadata server, determine the data server where the data block to be read is located. The data to be read can be read from the local data server where the data block to be read is located, or the data to be read can be read from another data server where the data block to be read is located through the local data server.

5. The data writing method according to claim 1, characterized in that, The process of allocating at least one data server for the data block to be written based on the data block name and the number of nodes of the data server includes: Based on the data block name of the data block to be written, the hash value of the data block to be written is calculated using a hash algorithm; Based on the hash value of the data block to be written and the number of data servers, at least one of the data servers is allocated to the data block to be written.

6. The data writing method according to claim 4, characterized in that, The step of setting a data block name for the data block to be read based on the logical position of the data to be read in the file containing the data to be read, the data fragment ID of the stored data containing the data to be read, and the logical position of the stored data in the file containing the data to be read includes: Based on the logical position of the data to be read in the file where the data to be read is located and the logical position of the stored data in the file where the data to be read is located, the logical position of the data block to be read in the data to be read is calculated; Based on the data fragment ID of the stored data where the data to be read is located and the logical position of the data block to be read in the data to be read, a data block name is set for the data block to be read.

7. A data writing method, characterized in that, The method is applied to a metadata server, which is communicatively connected to multiple data servers, and the data servers are communicatively connected to each other. The method includes: The system receives a query request sent by the data server after receiving a data write request. The data write request carries the file name of the file containing the data to be written, and the logical position of the data to be written in the file containing the data to be written. The query request carries the file name of the file containing the data to be written. The filename of the file containing the data to be written is used to assign a data fragment ID to the data to be written. The data fragment ID of the data to be written is sent to the data server, so that the data server can divide the received data to be written into blocks to obtain at least one data block to be written; The data server list is sent to the data server so that the data server allocates at least one data server for the data block to be written based on the name of the data block to be written and the number of nodes of the data server. The data server sends the data block to be written allocated to the local data server to the local data server, or sends the data block to be written allocated to other data servers through the local data server to the other data servers. The name of the data block to be written is set by the data server for the data block to be written based on the data fragment ID of the data to be written and the logical position of the data block to be written in the data to be written. The number of nodes of the data server is obtained based on the data server list. The other data servers are data servers other than the local data server.

8. The data writing method according to claim 7, characterized in that, The filename of the file containing the data to be written is called the data fragment ID allocated to the data to be written, including: Based on the file name of the file containing the data to be written, query the data fragment ID of the file that already stores data in the preset file index, which corresponds to the file name of the file containing the data to be written. A new data shard ID is assigned to the data to be written based on the data shard ID of the file whose file name corresponds to the file containing the data to be written.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the data writing method as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the data writing method as described in any one of claims 1 to 8.

Images