Data processing method and device based on distributed storage
By receiving data write requests in a distributed storage system, updating the index using container location descriptors, and adopting an append-only write approach, the write amplification problem caused by the pre-write log mechanism is solved, thus improving data read and write efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 云和恩墨(北京)信息技术有限公司
- Filing Date
- 2023-12-22
- Publication Date
- 2026-07-10
AI Technical Summary
In distributed storage systems, the write-before-log mechanism requires both data files and log files to be written to the storage area, causing write amplification and increasing computational and storage resource overhead.
By receiving data write requests in the distributed storage system, writing to multiple data containers, and updating the index using container location descriptors, the number of data writes is reduced. An append-only write method is used to merge data and update the state machine state.
It reduces the number of data writes, improves data read and write efficiency, solves the write amplification problem, and improves system performance.
Smart Images
Figure CN117742609B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed storage technology, and more specifically, to a data processing method and apparatus based on distributed storage. Background Technology
[0002] In distributed storage systems, to recover to the pre-failure state in the event of a system crash, a write-ahead logging (WAL) mechanism is typically used. Before writing data to its final location, a full log is written first. This way, after a system restart, replaying the log ensures the recovery of all successfully committed data. Write-ahead logging divides the host's write process into two phases: writing the log and applying the state machine. The log writing phase transforms random writes into sequential writes; when the log is applied, writing to the write cache returns a success message.
[0003] For traditional hard disk drives (HDDs), Write-Ahead Reading (WAL) significantly improves read and write performance while ensuring data persistence and atomicity. However, WAL also introduces the technical problem of write amplification, meaning that the same data can be written twice within a single node.
[0004] There is currently no effective solution to the above problems. Summary of the Invention
[0005] This application provides a data processing method and apparatus based on distributed storage to at least solve the technical problem of write amplification caused by the write-before-write logging mechanism in related technologies, where both log files and data files need to be written to the storage area.
[0006] According to one aspect of the embodiments of this application, a data processing method based on distributed storage is provided, comprising: a node in a distributed storage system receiving a data write request, wherein the write request includes: data to be written and a first index, wherein the first index is used to indicate the correspondence between the logical address of the data to be written and a first identifier, the first identifier being an identifier of the metadata of the data to be written; writing the data to be written into a target data container among multiple data containers to obtain written data and a container location descriptor of the written data, wherein the target data container includes multiple storage units, and the container location descriptor is used to record information of the target data container where the written data is located and information of the target storage unit where the written data is located; updating the first index based on the container location descriptor and the first identifier to obtain a second index; and updating the state of the state machine corresponding to the node according to the second index, wherein the second index is used to indicate the correspondence between the information of the storage unit where the written data is located and the container location descriptor in the metadata.
[0007] Optionally, each of the multiple storage units has the same capacity, x, where x is a natural number.
[0008] Optionally, writing the data to be written into a target data container among multiple data containers includes: if the capacity of the data to be written does not meet k*x, obtaining the offset of the data to be written relative to the target storage unit, where k is a positive integer; determining the initial data in the target storage unit based on the capacity of the target storage unit and the offset; merging the initial data with the data to be written to obtain the target data; and writing the target data into the target storage unit in the target data container.
[0009] Optionally, if the remaining space in multiple data containers is lower than a first preset threshold and / or according to a preset time interval, data in the first data container among the multiple data containers is deleted, wherein the first data container is any one of the multiple data containers other than the current data container.
[0010] Optionally, deleting data in a first data container among multiple data containers includes: determining the number of first storage units in each of the multiple data containers, wherein the first storage unit is a storage unit that does not correspond to a container location descriptor; if the number of first storage units in the first data container exceeds a second preset threshold, copying the data in the second storage unit to the current data container, wherein the second storage unit is a storage unit that corresponds to a container location descriptor; and deleting the first data container after copying the data in the second storage unit to the current data container.
[0011] Optionally, copying data from the second storage unit to the current data container includes: modifying the second index of the data to be copied to the current data container based on the information of the current data container where the data is located and the information of the current storage unit where the data is located.
[0012] Optionally, writing the data to be written to a target data container among multiple data containers includes: writing the data to be written to a target data container among multiple data containers by appending the data.
[0013] According to another aspect of the embodiments of this application, a data processing method based on distributed storage is also provided, comprising: a node in a distributed storage system receiving a data read request, wherein the data read request includes: data to be read and a logical address of the data to be read; determining the metadata of the data to be read based on the logical address of the data to be read through a second index, wherein the second index is used to indicate the correspondence between the logical address of the data to be read and a second identifier, the second identifier being an identifier of the metadata of the data to be read, the metadata of the data to be read including: the correspondence between the information of the storage unit where the data to be read is located and a container location descriptor, wherein the container location descriptor is used to record the information of the target data container where the data to be read is located and the information of the target storage unit where the data to be read is located; determining the information of the target data container where the data to be read is located and the information of the target storage unit where the data to be read is located based on the container location descriptor, and reading the data to be read based on the information of the target data container where the data to be read is located and the information of the target storage unit where the data to be read is located.
[0014] According to another aspect of the embodiments of this application, a data processing apparatus based on distributed storage is also provided, comprising: a receiving module, configured to receive a data write request, wherein the write request includes: data to be written and a first index, wherein the first index is used to indicate the correspondence between the logical address of the data to be written and a first identifier, the first identifier being an identifier of the metadata of the data to be written; a writing module, configured to write the data to be written into a target data container among multiple data containers to obtain written data and a container location descriptor of the written data, wherein the target data container includes multiple storage units, and the container location descriptor is used to record information of the target data container where the written data is located and information of the target storage unit where it is located; and an updating module, configured to update the first index based on the container location descriptor and the first identifier to obtain a second index, and update the state of the state machine corresponding to the node in the distributed storage system according to the second index, wherein the second index is used to indicate the correspondence between the information of the storage unit where the written data is located and the container location descriptor in the metadata.
[0015] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the storage medium including a stored program, wherein the program controls the device where the storage medium is located to execute the above-described data processing method based on distributed storage when it runs.
[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the program executes the above-described data processing method based on distributed storage during runtime.
[0017] In this embodiment, a node in a distributed storage system receives a data write request. The write request includes data to be written and a first index. The first index indicates the correspondence between the logical address of the data to be written and a first identifier, where the first identifier is the identifier of the metadata of the data to be written. The data to be written is written to a target data container among multiple data containers, resulting in written data and a container location descriptor for the written data. The target data container includes multiple storage units, and the container location descriptor records information about the target data container where the written data is located and information about the target storage unit where the written data is located. Based on the container location descriptor and the first identifier, the first index is updated to obtain a second index. And based on the second index, the state of the state machine corresponding to the node is updated. The second index is used to indicate the correspondence between the information of the storage unit where the data has been written in the metadata and the container location descriptor. By writing the data in the pre-write log into the target data container, the container location descriptor is obtained. When applying the log, the container location descriptor is carried, and the index information of the data block is updated using the container location descriptor. This allows the index information of the pre-write log and the data block to share the same data, thereby reducing the number of data writes and improving the technical effect of data read and write efficiency. This solves the technical problem of write amplification caused by the pre-write log mechanism, where both log files and data files need to be written to the storage area. Attached Figure Description
[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0019] Figure 1 This is a schematic diagram of a data processing method based on distributed storage, according to relevant technologies;
[0020] Figure 2 This is a flowchart of a data processing method based on distributed storage according to an embodiment of this application;
[0021] Figure 3 This is a schematic diagram of a data processing method based on distributed storage according to an embodiment of this application;
[0022] Figure 4 This is a flowchart of another data processing method based on distributed storage according to an embodiment of this application;
[0023] Figure 5 This is a schematic diagram of another data processing method based on distributed storage according to an embodiment of this application;
[0024] Figure 6This is a schematic diagram illustrating the deletion of data in a first data container according to an embodiment of this application;
[0025] Figure 7 This is a structural diagram of a data processing device based on distributed storage according to an embodiment of this application;
[0026] Figure 8 This is a hardware structure block diagram of a computer terminal for a data processing method based on distributed storage according to an embodiment of this application. Detailed Implementation
[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0029] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:
[0030] RAFT is a distributed consensus algorithm used to manage log replication and fault tolerance. RAFT ensures the existence of a leader in the system through an election mechanism. The leader is responsible for receiving client requests and replicating them to other nodes. The core of the RAFT algorithm is to divide the system state into three roles: leader, followers, and candidates, and to guarantee that there is only one leader in the system through election, heartbeat, and other mechanisms.
[0031] Write-Ahead Logging (WAL) is a commonly used logging mechanism in database systems. When using WAL, as the database system performs write operations, these operations are first recorded in a special log file before being applied to the database file. This mechanism ensures the stability and consistency of the database during write operations because even if unexpected situations occur during a write operation, the database system can still recover and repair itself using the records in the log file. WAL is typically used to ensure database durability and reliability, as well as improve database system performance.
[0032] In distributed storage systems, a Write-Ahead Log (WAL) mechanism is typically used to restore the system to its pre-failure state in the event of a system crash. Before writing data to its final location, a full log is written first. This way, after a system crash and restart, replaying the logs ensures the recovery of all successfully committed data. Figure 1 This is a schematic diagram of a data processing method based on distributed storage, according to related technologies. In a typical RAFT-based distributed storage system, the data writing process is shown in the figure:
[0033] 1. The host sends a read / write request to the master node. 2. The master node generates logs and replicates them to the standby nodes, appending log entries sequentially to the log storage area. 3. After a majority of nodes successfully write the logs, the master node applies the logs to the state machine: allocates space in the data storage area and updates the index information; writes the data to the write cache, at which point it can return a success message to the client; and flushes the write cache to the data storage area. 4. The master node notifies the standby nodes to apply the data to the state machine.
[0034] For traditional hard disk drives (HDDs), Write-Ahead Logs (WAL) significantly improves read and write performance while ensuring data persistence and atomicity. Specifically: Data persistence: The log contains complete data content. Once committed, the data is completely and persistently saved and will not be lost. Data atomicity: With relevant flushing strategies (uncommitted changes are not flushed to disk), when the system crashes and restarts, the system will redo the committed logs before providing services, thus preventing partial data updates. High performance: WAL divides the host's write process into two stages: writing to the log and applying the state machine. The log writing process transforms random writes into sequential writes; when applying the log, writing to the write cache returns success (without waiting for the write cache to be flushed to disk), turning synchronous writes into "asynchronous" writes.
[0035] On the other hand, WAL also brings write amplification. Within a single node, the same data will be written twice: the first time is when writing to the log, when all the data is written to the log area; the second time is when the log is applied to the state machine, when all the data is written to its final storage area in an overwrite manner.
[0036] The aforementioned high performance refers to mechanical hard drives. However, with the rapid popularization of non-volatile memory hard drives, the advantages of the above mechanism have gradually disappeared. Compared with mechanical hard drives, non-volatile memory hard drives have the following advantages: 1. High performance close to that of RAM; 2. The performance difference between random access and sequential access is not significant.
[0037] Therefore, for non-volatile hard disk drives, WAL offers almost no performance optimization, but its write amplification introduces additional overhead in terms of computation and storage resources, especially in distributed storage systems, where the write amplification factor increases significantly with the number of replicas.
[0038] In summary, in order to solve the above problems, this application provides relevant solutions, which are described in detail below.
[0039] According to an embodiment of this application, a method embodiment of a data processing method based on distributed storage is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0040] Figure 2 This is a flowchart of a data processing method based on distributed storage according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:
[0041] In step S202, a node in the distributed storage system receives a data write request, wherein the write request includes: data to be written and a first index, wherein the first index is used to indicate the correspondence between the logical address of the data to be written and a first identifier, and the first identifier is the identifier of the metadata of the data to be written.
[0042] In a distributed storage system, the master node receives data write requests from the host and forwards these requests to multiple backup nodes. In this system, the logical address of the data to be written is a unique identifier generated by the system to identify the data's location and storage information. The logical address may contain the identifier of the node or server where the data resides, as well as the data's specific location within that node. Using the logical address, the system can accurately locate the data's storage location and write it to the corresponding storage unit.
[0043] The metadata of the data to be written records information such as data type, size, storage location, version, permissions, creation time, modification time, access time, backup status, compression method, and encryption status. The metadata of the data to be written can be identified, for example, by its identification (ID).
[0044] Step S204: Write the data to be written into the target data container among multiple data containers to obtain the written data and the container location descriptor of the written data. The target data container includes multiple storage units, and the container location descriptor is used to record the information of the target data container where the written data is located and the information of the target storage unit where the written data is located.
[0045] For example, if the ID of the target data container where the written data is located is Container5, and the ID of the target storage unit where the written data is located is Block5, then the container location descriptor where the written data is located will record: Container5, Block5.
[0046] It should be noted that the physical storage space corresponding to a node can be divided into multiple target data containers, where the capacity of each target data container is a fixed value, for example, 4MB. The physical storage space of a node refers to the actual storage space owned by each node in a distributed storage system. This storage space can be a hard drive, solid-state drive, or other storage devices used to store data and files in the distributed system.
[0047] Furthermore, data to be written can only be written to the target data container via append write, and cannot be written to the target data container via overwrite write.
[0048] According to some alternative embodiments of this application, each of the multiple storage units has the same capacity, assumed to be x, where x is a natural number. During the process of writing data to the target data container via append-only writing, if the capacity of the data to be written does not meet k*x, the offset of the data to be written relative to the target storage unit is obtained, where k is a positive integer; based on the capacity of the target storage unit and the offset, the initial data in the target storage unit is determined; the initial data is merged with the data to be written to obtain the target data; and the target data is written to the target storage unit in the target data container.
[0049] For example, the data to be written is Chunk1, which has a length of 2KB and an offset of 2KB relative to its target memory location. During the append operation, initial data with an offset of 0 and a length of 2KB relative to the target memory location is read first. This initial data is then merged with the data to be written into a new data block, the target data. Finally, the target data is written to the target memory location within the target data container.
[0050] Step S206: Based on the container location descriptor and the first identifier, update the first index to obtain the second index, and update the state of the state machine corresponding to the node according to the second index. The second index is used to indicate the correspondence between the information of the storage unit where the data has been written in the metadata and the container location descriptor.
[0051] Preferably, the data processing method based on distributed storage in this embodiment is based on the RAFT algorithm. In a distributed storage system, each node has a state machine, which changes its state by executing commands in the log. RAFT ensures the consistency of the state machines of all nodes in the distributed system through log replication. Specifically, when a client sends a command to the distributed system, this command is appended to the system's log. Then, the RAFT algorithm ensures that this log is replicated by all nodes and applied to their respective state machines, thereby guaranteeing system consistency. In this way, regardless of which node processes the client request, all nodes' state machines will eventually reach a consistent state.
[0052] In some optional embodiments of this application, when the remaining space of multiple data containers is lower than a first preset threshold, and / or, according to a preset time interval, data in the first data container among the multiple data containers is deleted, wherein the first data container is any one of the multiple data containers other than the current data container, specifically including the following steps:
[0053] In each of the multiple data containers, the number of first storage units is determined, wherein the first storage unit is a storage unit that does not correspond to a container location descriptor; if the number of first storage units in the first data container exceeds a second preset threshold, the data in the second storage unit is copied to the current data container, wherein the second storage unit is a storage unit that corresponds to a container location descriptor; after copying the data in the second storage unit to the current data container, the first data container is deleted.
[0054] Preferably, copying data from the second storage unit to the current data container can be achieved by modifying the second index of the data to be copied to the current data container based on the information of the current data container where the data is located and the information of the current storage unit where the data is located.
[0055] Understandably, when data is overwritten in the same location or when a volume is deleted, the old data blocks will no longer have an index to reference them and will become garbage data. In this case, it is necessary to reclaim this space periodically.
[0056] Figure 6 This is a schematic diagram illustrating the deletion of data in a first data container according to an embodiment of this application, such as... Figure 6 As shown, scan each of the multiple data containers and perform the following operation for each data container:
[0057] Step S601: Count the number of all unreferenced storage units (first storage units). If the number does not exceed the second preset threshold, continue scanning the next data container; otherwise, proceed to step S602.
[0058] Step S602: Copy the data from the currently referenced storage unit (second storage unit) to the current data container and modify the corresponding index record;
[0059] Step S603: After completing the copying and modification of the data in all referenced storage units, delete the entire data container (first data container).
[0060] Figure 3 This is a schematic diagram of a data processing method based on distributed storage according to an embodiment of this application, as shown below. Figure 3 As shown, when writing to the log, the log description information and the log data (Chunk) are separated.
[0061] Step S301: Use a data container (Container) to manage physical storage space. The size of the container is fixed, and the physical storage space can be divided into multiple containers. Containers can only be appended to and cannot be overwritten. The storage space in a container can be further divided into multiple (e.g., 1024) fixed-size (e.g., 4K) blocks.
[0062] In step S302, the data portion of the log is directly written into the Container. After successful writing, a Container Position Descriptor (CPD) is returned, where the CPD is used to locate which block of which Container the data is stored in.
[0063] In step S303, when a node applies a log, it no longer carries data, but instead carries the CPD.
[0064] Step S304: Update the CPD of the corresponding Block in the Chunk Meta using the CPD carried in the request.
[0065] Based on the above steps, by writing the data in the pre-write log into the target data container, a container location descriptor is obtained. When applying the log, the container location descriptor is carried, and the index information of the data block is updated using the container location descriptor. This allows the pre-write log and the index information of the data block to share the same data, thereby reducing the number of data writes and achieving the technical effect of improving data read and write efficiency.
[0066] Figure 4 This is a flowchart of another data processing method based on distributed storage according to an embodiment of this application, such as... Figure 4 As shown, the method includes the following steps:
[0067] In step S402, a node in the distributed storage system receives a data read request, wherein the data read request includes: the data to be read and the logical address of the data to be read.
[0068] Step S404: Based on the logical address of the data to be read, determine the metadata of the data to be read through the second index. The second index is used to indicate the correspondence between the logical address of the data to be read and the second identifier. The second identifier is the identifier of the metadata of the data to be read. The metadata of the data to be read includes: the correspondence between the information of the storage unit where the data to be read is located and the container location descriptor. The container location descriptor is used to record the information of the target data container where the data to be read is located and the information of the target storage unit where the data to be read is located.
[0069] Step S406: Based on the container location descriptor, determine the information of the target data container where the data to be read is located and the information of the target storage unit where it is located, and read the data to be read based on the information of the target data container where the data to be read is located and the information of the target storage unit where it is located.
[0070] Figure 5 This is a schematic diagram of another data processing method based on distributed storage according to an embodiment of this application, such as... Figure 5 As shown, the method includes the following steps:
[0071] In step S501, a node in the distributed storage system receives a data read request, which carries the logical address of the data to be read, such as the ID of the sector (Vol_ID) where the data to be read is located and the ID of the block (Chunk_ID) where the data to be read is located. Based on the ID of the sector and the ID of the block where the data to be read is located, a hash algorithm is used to determine the ID of the bucket where the data to be read is located.
[0072] Step S502: Using the ID of the bucket where the data to be read resides as the key value of the Chunk index, the metadata of the data to be read is determined. The Chunk index includes the following information: 1. A unique identifier for the data block: used to uniquely identify each data block for location and access within the storage system. 2. Location information of the storage node: records the node location where each data block is stored, enabling quick location and access to the data block when needed. 3. Metadata information of the data block: includes metadata information such as the data block size, creation time, and permissions, used to manage and control access to and use of the data block.
[0073] Step S503: The metadata includes the mapping relationship between Block and CPD, and the CPD is obtained through this mapping relationship.
[0074] In step S504, the CPD can be used to accurately locate which block of which container the data is actually stored in.
[0075] Step S505: Read data from the specified container and return the data to the host.
[0076] Figure 7 This is a structural diagram of a data processing device based on distributed storage according to an embodiment of this application, such as... Figure 7 As shown, the device includes:
[0077] The receiving module 70 is used to receive a data write request, wherein the write request includes: data to be written and a first index, wherein the first index is used to indicate the correspondence between the logical address of the data to be written and a first identifier, and the first identifier is the identifier of the metadata of the data to be written.
[0078] The writing module 72 is used to write the data to be written into a target data container among multiple data containers, and to obtain the written data and the container location descriptor of the written data. The target data container includes multiple storage units, and the container location descriptor is used to record the information of the target data container where the written data is located and the information of the target storage unit where it is located.
[0079] The update module 74 is used to update the first index based on the container location descriptor and the first identifier to obtain the second index, and to update the state of the state machine corresponding to the node in the distributed storage system according to the second index. The second index is used to indicate the correspondence between the information of the storage unit where the data has been written in the metadata and the container location descriptor.
[0080] It should be noted that the above Figure 7 The modules in the above can be program modules (e.g., a set of program instructions that implement a specific function) or hardware modules. For the latter, they can be represented in the following forms, but are not limited to these: each of the above modules is represented by a processor, or the functions of each of the above modules are implemented by a processor.
[0081] It should be noted that, Figure 7 Preferred embodiments of the shown examples can be found in [reference needed]. Figure 2 The relevant descriptions of the embodiments shown will not be repeated here.
[0082] Figure 8 A hardware block diagram of a computer terminal for implementing a data processing method based on distributed storage is shown. Figure 8 As shown, the computer terminal 80 may include one or more processors 802 (shown as 802a, 802b, ..., 802n in the figure) 802 (processor 802 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 804 for storing data, and a transmission module 806 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 8 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, the computer terminal 80 may also include... Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown.
[0083] It should be noted that the aforementioned one or more processors 802 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 80. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).
[0084] The memory 804 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data processing method based on distributed storage in this embodiment. The processor 802 executes various functional applications and data processing by running the software programs and modules stored in the memory 804, thereby realizing the aforementioned data processing method based on distributed storage. The memory 804 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 804 may further include memory remotely located relative to the processor 802, and these remote memories can be connected to the computer terminal 80 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0085] The transmission module 806 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 80. In one example, the transmission module 806 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 806 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.
[0086] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 80.
[0087] It should be noted here that, in some optional embodiments, the above... Figure 8 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 8 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.
[0088] It should be noted that, Figure 8 The computer terminal shown is used to execute Figure 2 The data processing method based on distributed storage shown above also applies to this electronic device, and will not be repeated here.
[0089] This application also provides a non-volatile storage medium, which includes a stored program, wherein the program, when running, controls the device where the storage medium is located to execute the above-described data processing method based on distributed storage.
[0090] A non-volatile storage medium performs the following functions: A node in a distributed storage system receives a data write request, wherein the write request includes: data to be written and a first index, wherein the first index is used to indicate the correspondence between the logical address of the data to be written and a first identifier, and the first identifier is the identifier of the metadata of the data to be written; the data to be written is written to a target data container among multiple data containers to obtain written data and a container location descriptor of the written data, wherein the target data container includes multiple storage units, and the container location descriptor is used to record the information of the target data container where the written data is located and the information of the target storage unit where the written data is located; based on the container location descriptor and the first identifier, the first index is updated to obtain a second index, and the state of the state machine corresponding to the node is updated according to the second index, wherein the second index is used to indicate the correspondence between the information of the storage unit where the written data is located and the container location descriptor in the metadata.
[0091] This application also provides an electronic device, including a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the above-described data processing method based on distributed storage.
[0092] The processor is used to run a program that performs the following functions: a node in a distributed storage system receives a data write request, wherein the write request includes: data to be written and a first index, wherein the first index is used to indicate the correspondence between the logical address of the data to be written and a first identifier, and the first identifier is the identifier of the metadata of the data to be written; the data to be written is written to a target data container among multiple data containers to obtain written data and a container location descriptor of the written data, wherein the target data container includes multiple storage units, and the container location descriptor is used to record the information of the target data container where the written data is located and the information of the target storage unit where the written data is located; based on the container location descriptor and the first identifier, the first index is updated to obtain a second index, and the state of the state machine corresponding to the node is updated according to the second index, wherein the second index is used to indicate the correspondence between the information of the storage unit where the written data is located and the container location descriptor in the metadata.
[0093] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0094] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0095] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0096] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0097] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0098] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0099] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A data processing method based on distributed storage, characterized in that, include: A node in a distributed storage system receives a data write request, wherein the write request includes: data to be written and a first index, wherein the first index is used to indicate the correspondence between the logical address of the data to be written and a first identifier, and the first identifier is an identifier of the metadata of the data to be written; The data to be written is written into a target data container among multiple data containers to obtain written data and the container location descriptor of the written data. The target data container includes multiple storage units, and the container location descriptor is used to record information about the target data container where the written data is located and information about the target storage unit where the written data is located. Based on the container location descriptor and the first identifier, the first index is updated to obtain a second index, and the state of the state machine corresponding to the node is updated according to the second index, wherein the second index is used to indicate the correspondence between the information of the storage unit where the written data is located and the container location descriptor in the metadata.
2. The method according to claim 1, characterized in that, Each of the plurality of storage units has the same capacity, x, where x is a natural number.
3. The method according to claim 2, characterized in that, Writing the data to be written into a target data container among multiple data containers includes: If the capacity of the data to be written does not meet k*x, obtain the offset of the data to be written relative to the target storage unit, where k is a positive integer; Based on the capacity of the target storage unit and the offset, determine the initial data in the target storage unit; The initial data is merged with the data to be written to obtain the target data; The target data is written into the target storage unit in the target data container.
4. The method according to claim 1, characterized in that, The method further includes: deleting data in a first data container among the plurality of data containers when the remaining space of the plurality of data containers is lower than a first preset threshold and / or at preset time intervals, wherein the first data container is any one of the plurality of data containers other than the current data container.
5. The method according to claim 4, characterized in that, Deleting data from the first data container among the plurality of data containers includes: In each of the plurality of data containers, the number of first storage units is determined, wherein the first storage unit is a storage unit that does not correspond to the container location descriptor; If the number of the first storage units in the first data container exceeds a second preset threshold, the data in the second storage unit is copied to the current data container, wherein the second storage unit is a storage unit that corresponds to the container location descriptor; After copying the data from the second storage unit to the current data container, the first data container is deleted.
6. The method according to claim 5, characterized in that, Copying data from the second storage unit to the current data container includes: modifying the second index of the data to be copied to the current data container based on the information of the current data container where the data is located and the information of the current storage unit where the data is currently located.
7. The method according to claim 1, characterized in that, Writing the data to be written to a target data container among multiple data containers includes: writing the data to be written to the target data container among multiple data containers by appending the data.
8. A data processing method based on distributed storage, characterized in that, include: In a distributed storage system, a node receives a data read request, wherein the data read request includes: the data to be read and the logical address of the data to be read; Based on the logical address of the data to be read, the metadata of the data to be read is determined through a second index. The second index is used to indicate the correspondence between the logical address of the data to be read and a second identifier. The second identifier is the identifier of the metadata of the data to be read. The metadata of the data to be read includes: the correspondence between the information of the storage unit where the data to be read is located and the container location descriptor. The container location descriptor is used to record the information of the target data container where the data to be read is located and the information of the target storage unit where the data to be read is located. Based on the container location descriptor, the information of the target data container where the data to be read is located and the information of the target storage unit where it is located are determined, and the data to be read is read based on the information of the target data container where the data to be read is located and the information of the target storage unit where it is located.
9. A data processing device based on distributed storage, characterized in that, include: A receiving module is configured to receive a data write request, wherein the write request includes: data to be written and a first index, wherein the first index is used to indicate the correspondence between the logical address of the data to be written and a first identifier, and the first identifier is an identifier of the metadata of the data to be written; The writing module is used to write the data to be written into a target data container among multiple data containers, and to obtain the written data and the container location descriptor of the written data. The target data container includes multiple storage units, and the container location descriptor is used to record the information of the target data container where the written data is located and the information of the target storage unit where it is located. An update module is configured to update the first index based on the container location descriptor and the first identifier to obtain a second index, and update the state of the state machine corresponding to the node in the distributed storage system according to the second index, wherein the second index is used to indicate the correspondence between the information of the storage unit where the written data is located and the container location descriptor in the metadata.
10. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein, when the program is executed, it controls the device where the non-volatile storage medium is located to perform the data processing method based on distributed storage as described in any one of claims 1 to 8.
11. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the data processing method based on distributed storage as described in any one of claims 1 to 8.