Data management method and apparatus, device, and storage medium thereof
By storing metadata in shared memory and having the storage end handle metadata updates and read/write requests, the problems of poor file system performance and low cache hit rate in existing technologies are solved, achieving efficient database file operations.
Patent Information
- Application Number
- PCT/CN2023/098922
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-02
- Filing Date
- 2023-06-07
- Publication Date
- 2024-12-05
AI Technical Summary
In clustered databases based on a shared storage architecture, existing file system metadata management methods suffer from poor performance and low cache hit rates. This is especially true in file system server-side management and client-side caching schemes, where read operations require frequent client-server round trips and the cache hit rate is insufficient.
All metadata is stored in shared memory. The storage side initializes and generates shared memory, and handles metadata updates and read/write requests through the storage side. Clients directly access shared memory to perform file operations, avoiding frequent server round trips and cache misses.
It improves database read/write efficiency, reduces network overhead, enhances metadata hit rate, and achieves high-performance file system.
Smart Images

Figure CN2023098922_05122024_PF_FP_ABST
Abstract
Description
Data management method, device, equipment and storage medium thereof Technical Field
[0001] The present application relates to the field of data management, and in particular to a data management method, apparatus, device, and storage medium thereof. Background Art
[0002] In a cluster database based on a shared storage architecture, database instances run on multiple physical hosts. These instances concurrently read and write the same data, including but not limited to database control files, data files, and log files. During operation, these instances concurrently read and write the same file, and may also create and delete files and organize files in a directory format. This often requires a parallel file system to manage the disks divided on the shared storage and provide a directory file interface for the database to use. To meet the above database requirements, a parallel file system, such as GPFS, is required. However, the performance of these general-purpose file systems is less than satisfactory.
[0003] In a parallel file system, there are two main methods for managing file system metadata: file system server-side management only and file system server-side management + client-side caching.
[0004] Among them, in the file system server-only management: that is, only the file system server manages metadata information; the client does not cache metadata information; the file system master instance is responsible for updating metadata and synchronizing such updates to other file system non-master instances; when the business side needs to read user data, it needs to forward the request to the file system server, and after the file system server successfully reads, it returns it to the business side; each read requires a round trip between the client and the server, and the performance is very poor.
[0005] In file system server management + client cache: This is an improvement based on file system server management only; that is, the file system server manages metadata information, but the client manages the cache of some metadata; when the business side reads user data, if the metadata information hits the cache, the file's location on the shared disk array can be calculated and directly read and written; although this improves read and write efficiency, the cache still has a hit rate problem.
[0006] Summary of the Invention
[0007] In view of the above problems, the present application is proposed to provide a data management method, apparatus, device and storage medium thereof that overcome the above problems or at least partially solve the above problems, including:
[0008] A data management method, the method involving a storage end and a client end, wherein the storage end is used to process metadata, the processing including update processing and read and write processing, the method comprising:
[0009] The storage end receives a connection request sent by the client, determines a key value corresponding to a shared memory according to the connection request, and sends the key value to the client, wherein the shared memory is generated by initialization processing of the storage end;
[0010] When the storage end receives the metadata update request sent by the client, the storage end determines the metadata to be updated based on the metadata update request, and allocates and processes the metadata to be updated to the shared memory, wherein the metadata to be updated includes directories, files, and file partition tables.
[0011] Furthermore, the step of generating the shared memory by initialization processing of the storage end includes:
[0012] Determine the target shared disk based on the preset configuration information.
[0013] When it is detected that the target shared disk is not formatted, the storage end formats the target shared disk according to the preset configuration information, and initializes the target shared disk to obtain BOOT information;
[0014] The storage end determines the number information according to the BOOT information;
[0015] The storage end constructs a corresponding number of to-be-initialized shared memories according to the number of the number information, wherein at least one of the to-be-initialized shared memories is used to store manager information;
[0016] The storage end determines the disk location corresponding to each shared memory to be initialized according to the manager information;
[0017] The storage end adds the initial metadata to the corresponding shared memory to be initialized in the target disk according to the disk position, and marks the initialized shared memory as the shared memory after the initial metadata is added.
[0018] Furthermore, the step of determining metadata to be updated based on the metadata update request and allocating the metadata to be updated to the shared memory, wherein the metadata to be updated includes directories, files, and file partition tables, comprises:
[0019] The storage end determines whether there is a first target data block corresponding to the metadata to be updated according to the metadata to be updated;
[0020] When the first target data block does not exist, the storage end determines whether there is an unused data block in the current shared memory;
[0021] When the current shared memory does not contain the unused data block, the storage end generates a shared memory construction request;
[0022] The storage end constructs a target shared memory according to the shared memory construction request;
[0023] The storage end determines a second target data block in the target shared memory for storing the metadata to be updated, and writes the metadata to be updated into the second target data block.
[0024] An embodiment of the present invention further provides a data management method, the method involving a storage end and a client end, wherein the storage end is used to process metadata, the processing including update processing and read and write processing, and the method includes:
[0025] When detecting that the database instance is started, the client generates a connection request and sends the connection request to the storage end;
[0026] When the client receives the key value sent by the storage end, the client connects to the shared memory according to the key value;
[0027] When the client has a task to be updated, the client generates a metadata update request according to the metadata to be updated in the task to be updated, and sends the metadata update request to the storage end;
[0028] When the client has a service data read / write request, the client reads the target file according to the service data read / write request and the target metadata in the shared memory.
[0029] Furthermore, the step of reading the target file according to the business data read / write request and the target metadata in the shared memory includes:
[0030] The client determines the target metadata according to the read / write request;
[0031] The client determines the target disk location according to the target metadata;
[0032] The client obtains the target file according to the target disk location.
[0033] Furthermore, the step of determining the target metadata according to the read / write request includes:
[0034] The client obtains all metadata from the shared memory according to the read / write request;
[0035] The client determines all data blocks corresponding to all metadata according to all metadata;
[0036] When all the data blocks do not exist in the existing shared memory, the client queries the target manager information of the existing shared memory, and determines whether there is a new shared memory to be loaded according to the target manager information;
[0037] When the new shared memory needs to be loaded, the client obtains the slot information and the target key value in the target manager information to generate the new shared memory;
[0038] The client determines a third target data block according to the new shared memory;
[0039] The client determines the target metadata according to the third target data block.
[0040] An embodiment of the present invention further provides a data management device, the device involving a storage end and a client end, wherein the storage end is used to process metadata, the processing including update processing and read and write processing, and the device includes:
[0041] A first sending module is configured to receive, at the storage end, a connection request sent by the client, determine a key value corresponding to a shared memory according to the connection request, and send the key value to the client, wherein the shared memory is generated by initialization processing by the storage end;
[0042] The first allocation module is used to determine the metadata to be updated based on the metadata update request sent by the client when the storage end receives the metadata update request, and allocate the metadata to be updated to the shared memory, wherein the metadata to be updated includes directories, files and file partition tables.
[0043] An embodiment of the present invention further provides a data management device, the device involving a storage end and a client end, wherein the storage end is used to process metadata, the processing including update processing and read and write processing, and the device includes:
[0044] A second sending module is configured to, when detecting that the database instance is started, generate a connection request by the client and send the connection request to the storage end;
[0045] A first connection module is configured to connect the client to the shared memory according to the key value when the client receives the key value sent by the storage end;
[0046] A third sending module is configured to, when the client has a task to be updated, generate a metadata update request based on the metadata to be updated in the task to be updated, and send the metadata update request to the storage end;
[0047] The first reading module is configured to, when the client has a business data read / write request, enable the client to read a target file according to the business data read / write request and the target metadata in the shared memory.
[0048] An embodiment of the present invention further provides a computer device comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the data management method described above.
[0049] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data management method described above are implemented.
[0050] This application has the following advantages:
[0051] In an embodiment of the present application, compared with the prior art in which "each read requires a round trip between the storage end and the server end, the performance is very poor, and the cache still has a hit rate problem", the present application provides a solution of "a data management method", specifically: "The storage end receives the connection request sent by the client, and determines the key value corresponding to the shared memory based on the connection request, and sends the key value to the client, wherein the shared memory is generated by the initialization processing of the storage end; when the storage end receives the metadata update request sent by the client, the storage end determines the metadata to be updated based on the metadata update request, and allocates the metadata to be updated to the shared memory, wherein the metadata to be updated includes directories, files and file partition tables." The "storage side" solves the problem of "each read requires a round trip between the client and the server, the performance is very poor, and the cache still has a hit rate", and achieves the effect of "the database can determine the location of the data file on the shared disk array through the metadata information in the shared memory, and then perform direct reading and writing, which greatly improves the database read and write efficiency. The database and the file system share memory, which reduces memory usage and improves the database side's metadata hit rate compared to the method of the database caching file system metadata or caching part of the metadata; the database uses shared memory for files. Compared with the database throwing read and write requests and metadata update requests to the file system instance for processing, it not only improves read and write efficiency but also reduces network overhead." BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0053] FIG1 is a flowchart of a data management method according to an embodiment of the present application;
[0054] FIG2 is a flowchart of a data management method according to an embodiment of the present application;
[0055] FIG3 is a structural block diagram of a data management device provided by an embodiment of the present application;
[0056] FIG4 is a structural block diagram of a data management device provided by an embodiment of the present application;
[0057] FIG5 is a schematic structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0058] To make the objectives, features, and advantages of this application more readily apparent, the present application is further described below in conjunction with the accompanying drawings and specific embodiments. It is apparent that the embodiments described are only a portion of the embodiments of this application, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments in this application without inventive effort are also within the scope of protection of this application.
[0059] The inventors have found through analysis of the prior art that, in a parallel file system, there are two main methods for managing file system metadata: file system server management only and file system server management + client cache.
[0060] 1. File system server-only management: This means that only the file system server manages metadata information. Clients do not cache metadata information. The file system master instance is responsible for updating metadata and synchronizing these updates to other non-master instances. When the business side needs to read user data, it forwards the request to the file system server. After the file system server successfully reads the data, it returns it to the business side. Each read requires a round trip between the client and the server, resulting in very poor performance.
[0061] 2. File system server management + client caching: This is an improvement over file system server-only management. Specifically, the file system server manages metadata, while the client manages a cache of some metadata. When the business side reads user data, if the metadata information is matched in the cache, the file's location on the shared disk array is calculated and read or written directly. Compared to file system server-only management, this significantly improves read and write efficiency; however, the cache has a hit rate issue.
[0062] The following prior art was retrieved:
[0063] Prior Art 1: Application number is "200910085706.1", announcement date is "May 16, 2012", patent name is "Management and allocation method of resources in parallel file system", in which, all metadata are stored on the file system server, and the client caches metadata. The following is an analysis of its shortcomings through the reading process
[0064] Reading process:
[0065] 1. When accessing a file for read, the client first checks whether the metadata of the corresponding file is cached locally. If so, it jumps to 3.
[0066] 2. If the client cache does not hit the target, the interaction module will be used to pull metadata from the server and update it to the client's local cache.
[0067] 3. At this time, the client can obtain file partition information through the local cache and read data.
[0068] Therefore, the prior art 1 adopts the method of caching metadata on the client side, which has a certain hit rate problem. However, the technical solution of this application stores all metadata in the shared memory, which eliminates the miss rate problem.
[0069] Prior Art 2: Application number is "201210385489.X", announcement date is "February 27, 2013", and the patent name is "A method for optimizing disk database performance using in-memory database technology". The invention provides a method for optimizing disk database performance using in-memory database technology. The method is implemented by installing and deploying an in-memory database and middleware application on a server equipped with a large memory, and then caching hot data stored in the disk database, including read-only data and readable and writable data, into the memory. The corresponding data synchronization method is configured to automatically synchronize the changed data in the memory to the disk database to ensure data reliability. By utilizing in-memory database technology, the hot data stored on the disk is cached in the memory. At the same time, the application is modified to configure the original request for data on the disk to request data in the memory. This eliminates the I / O bottleneck in the traditional disk database system and greatly improves the system performance.
[0070] Prior Art 2 caches hot data from the disk database into an in-memory database, effectively adding a layer of cache between the disk database and the business side, reducing the disk database's I / O bottleneck. The present application's technical solution, on the other hand, places all metadata in "shared memory," allowing the business side to calculate the file's actual location in the SAN using the metadata and perform direct I / O operations. While Prior Art 2 uses in-memory, the present application uses shared memory, which is implemented differently. Furthermore, the two approaches do not address the same type of problem.
[0071] By solving the problems of the existing technologies 1 and 2 and combining them with the problems of the file system metadata management method, this application proposes a solution to store all metadata in shared memory. The file system server and client can see the same metadata information. When the business side reads user data, it can directly read the metadata information in the shared memory, calculate the location of the file on the shared disk array, and perform direct I / O. Because the metadata information seen by the client and the server is consistent, there is no miss problem. The technical solution of this application is described in detail as follows:
[0072] 1 , there is shown a flowchart of the steps of a data management method provided by an embodiment of the present application;
[0073] A data management method, the method involving a storage end and a client end, wherein the storage end is used to process metadata, the processing including update processing and read and write processing, the method comprising:
[0074] S110, the storage end receives a connection request sent by the client, determines a key value corresponding to a shared memory according to the connection request, and sends the key value to the client, wherein the shared memory is generated by initialization processing by the storage end;
[0075] S120. When the storage end receives the metadata update request sent by the client, the storage end determines the metadata to be updated based on the metadata update request, and allocates the metadata to be updated to the shared memory, wherein the metadata to be updated includes directories, files, and file partition tables.
[0076] In an embodiment of the present application, compared with the prior art in which "each read requires a round trip between the storage end and the server end, the performance is very poor, and the cache still has a hit rate problem", the present application provides a solution of "a data management method", specifically: "The storage end receives the connection request sent by the client, and determines the key value corresponding to the shared memory based on the connection request, and sends the key value to the client, wherein the shared memory is generated by the initialization processing of the storage end; when the storage end receives the metadata update request sent by the client, the storage end determines the metadata to be updated based on the metadata update request, and allocates the metadata to be updated to the shared memory, wherein the metadata to be updated includes directories, files and file partition tables." The "storage side" solves the problem of "each read requires a round trip between the client and the server, the performance is very poor, and the cache still has a hit rate." The database can determine the location of the data file on the shared disk array through the metadata information in the shared memory, and then perform direct reading and writing, which greatly improves the database read and write efficiency. The database and the file system share memory, which reduces memory usage and improves the database side's metadata hit rate compared to the database caching file system metadata or caching partial metadata. The database uses shared memory for files. Compared with the database throwing read and write requests and metadata update requests to the file system instance for processing, it not only improves read and write efficiency but also reduces network overhead."
[0077] Next, a data management method in this exemplary embodiment will be further described.
[0078] It should be noted that file operations are divided into metadata update processing (such as creating a file, extending the file size, etc.) and read and write processing. These two types of processing are separated. That is, when writing a file, the file size cannot be expanded, the file attributes cannot be modified, etc.
[0079] It should be noted that the key value is sent to the client via a heartbeat packet.
[0080] It's important to note that a heartbeat packet is a custom command word that periodically communicates the status of the storage device and the server. It's sent at regular intervals, similar to a heartbeat, hence the name. Networks use sockets to send and receive data, but if a socket is disconnected, sending and receiving data can be problematic. To determine whether a socket is still usable, a heartbeat mechanism is required. The Transmission Control Protocol (TCP) already implements this mechanism. If a heartbeat is configured, the TCP will send a set number of heartbeats (e.g., two) within a specified time interval (e.g., three seconds). This information does not affect the custom protocol. A heartbeat is a custom structure (a heartbeat packet or frame) sent periodically to let the other party know it's online and ensure the link is valid. A heartbeat packet is simply a simple message sent by the storage device to the server to indicate it's still online. The code simply sends a fixed message to the server every few minutes, and the server responds with a fixed message. If the server doesn't receive a message from the storage device within a few minutes, the storage device is considered disconnected. For example, if some communication software isn't used for a long time, knowing whether it's online or offline requires heartbeat packets, which are sent and received regularly. The sender can be either the client or the server, depending on which implementation is more convenient and reasonable. Typically, this is the storage client. The server can also periodically poll and send heartbeats. Heartbeat packets are so called because, like heartbeats, they are sent at fixed intervals to let the server know that the storage client is still alive. This is done to maintain a persistent connection. There are no specific requirements for the contents of these packets, but they are typically very small or contain only an empty header.
[0081] As an example, there are two ways for a database instance to obtain the key value corresponding to a newly applied or newly constructed shared memory:
[0082] The first method: A heartbeat is maintained between the database instance and the file system instance. The key value of the shared memory can be returned to the database instance through the heartbeat packet. When the database instance finds that the key value has been added, it knows that the new application has newly built the shared memory and can connect.
[0083] The second method: When the database instance reads or writes a file, if it finds that the file partition does not exist in the current shared memory, it queries the file system for the key value of the shared memory, obtains the key value, and then connects.
[0084] In a specific implementation, the storage end receives a connection request sent by the client, obtains a key value of the shared memory through the connection request, and sends the key value to the client.
[0085] In one embodiment of the present invention, after receiving a metadata update request, the file system instance allocates metadata information in a shared memory according to preset rules; the file system instance service synchronizes the metadata modification to other file system instances through an inter-cluster synchronization processing thread.
[0086] In one embodiment of the present invention, when the database reads and writes a file, it directly calculates the actual storage location of the file in the shared disk array through the metadata information in the shared memory, and then directly performs the read and write operations.
[0087] In one embodiment of the present invention, the step of generating the shared memory by initialization processing on the storage end includes:
[0088] Determine the target shared disk based on the preset configuration information.
[0089] When it is detected that the target shared disk is not formatted, the storage end formats the target shared disk according to the preset configuration information, and initializes the target shared disk to obtain BOOT information;
[0090] The storage end determines the number information according to the BOOT information;
[0091] The storage end constructs a corresponding number of to-be-initialized shared memories according to the number of the number information, wherein at least one of the to-be-initialized shared memories is used to store manager information;
[0092] The storage end determines the disk location corresponding to each shared memory to be initialized according to the manager information;
[0093] The storage end adds the initial metadata to the corresponding shared memory to be initialized in the target disk according to the disk position, and marks the initialized shared memory as the shared memory after the initial metadata is added.
[0094] It should be noted that the shared disk is a disk array. A disk array is a disk group that combines multiple independent hard disks in different ways to provide higher storage performance than a single hard disk and provides data backup technology. The different ways of forming a disk array are called magnetic array levels. The function of data backup is to use the backup information to restore the damaged data once the user data is damaged, thereby ensuring the security of the user data. To the user, the disk group is like a hard disk, and the user can partition and format it. It has the two major characteristics of high speed and security.
[0095] It should be noted that the BOOT information is boot information.
[0096] It should be noted that the size of shared memory is not fixed. As the business continues to run and the metadata continues to grow, the current shared memory can no longer meet the demand. At this time, another piece of shared memory of the same size will be requested and constructed, and synchronized to other file system instances. The number of shared memory allocated is determined by the configuration file, that is, the preset configuration information.
[0097] It should be noted that the number information refers to the number of shared memories stored by the shared memory manager, ie, the manager information.
[0098] It should be noted that the manager information refers to the shared memory manager, which manages these shared memories of the same size through the shared memory manager; the shared memory manager itself is also a shared memory, which has multiple slots for recording the shared memory that has been applied for in the system and the location of its data on the target shared disk. The data recorded in the shared memory will eventually be stored on the shared disk array; each piece of shared memory will eventually be stored on the fixed hard disk through the redo log file; the number of slots is given by the configuration file, that is, the preset configuration information, when the file system is started for the first time.
[0099] As an example, when determining whether the target shared disk is formatted, if it is formatted, the BOOT information can be directly obtained.
[0100] As an example, the shared memory to be initialized does not store the initial metadata, and the corresponding initial metadata needs to be added to the shared memory to be initialized through the disk location before the shared memory is formed.
[0101] As an example, when the file system starts, it first applies for a shared memory to store metadata information.
[0102] As an example, at least one shared memory to be initialized is included for storing manager information, and a shared memory to be initialized is constructed by applying first to store manager information of a shared memory manager.
[0103] For example, shared memory is not allocated all at once; instead, a 32MB block of shared memory is allocated first, which is then divided into 4KB blocks. When using a block, a block is first allocated from the shared memory, and then the metadata is stored in that block.
[0104] In a specific implementation, after loading the configuration file, i.e., the preset configuration information, it is determined whether the target shared disk is formatted. If not, the target shared disk needs to be formatted. The number of metadata of the shared memory is determined according to the configuration file, i.e., the preset configuration information, which is used to determine the number of slots of the shared memory manager. After the formatting process is completed, the BOOT information is obtained in the target shared disk, and the shared memory is generated through the BOOT information.
[0105] In a specific implementation, first, a block of shared memory to be initialized is applied for and initialized. The shared memory to be initialized is used to store manager information. According to the number of shared memories stored by the shared memory manager in the BOOT information, the shared memory is applied for and constructed, wherein the size of each shared memory block is 32M. The disk location corresponding to each shared memory block is determined according to the shared memory manager, initial metadata is loaded from the corresponding disk location, and the initial metadata is flushed into the shared memory, i.e., added to the shared memory.
[0106] In one embodiment of the present invention, the step of determining metadata to be updated based on the metadata update request and allocating the metadata to be updated to the shared memory, wherein the metadata to be updated includes directories, files, and file partition tables, includes:
[0107] The storage end determines whether there is a first target data block corresponding to the metadata to be updated according to the metadata to be updated;
[0108] When the first target data block does not exist, the storage end determines whether there is an unused data block in the current shared memory;
[0109] When the current shared memory does not contain the unused data block, the storage end generates a shared memory construction request;
[0110] The storage end constructs a target shared memory according to the shared memory construction request;
[0111] The storage end determines a second target data block in the target shared memory for storing the metadata to be updated, and writes the metadata to be updated into the second target data block.
[0112] It should be noted that the current shared memory is not the target shared memory, and the target shared memory is newly constructed.
[0113] It should be noted that after the file system is integrated with the database, the database creates directories, files, modifies file sizes, and other metadata modification operations through the file system storage end, and sends them to the file system instance for processing through UDS (the UDS protocol is ISO14229, which is Unified Diagnostic Services, a standardized standard for diagnostic services). After processing is completed, the directories, files, and file partition tables are updated to the shared memory in a certain organizational form. In a specific embodiment, metadata updates involve creating directories, modifying directories, deleting directories, creating files, deleting files, extending files, truncating files, etc. These operations involving metadata modification require the file system storage end to send the request to the server end through UDS for processing. After processing is completed, the metadata is written to the shared memory and passed to other file system nodes in the cluster.
[0114] As an example, when a metadata update request is received, it is first determined whether the first target data block to which the metadata to be updated belongs already exists. If so, the metadata to be updated is directly written to the first target data block.
[0115] As an example, the storage end determines whether there is an unused data block in the current shared memory; when the unused data block exists in the current shared memory, an unused data block is requested from the current shared memory and recorded in a redo log file.
[0116] In a specific implementation, when a metadata update request is received, it is first determined whether the first target data block to which the metadata to be updated belongs already exists. If not, it is checked whether there are any unused data blocks in the current shared memory. If not, a request for building a shared memory is generated. By building a shared memory request, a slot is requested in the metadata shared memory management, and a target shared memory is requested and built. The size of the target shared memory is 32M, initialized as shared memory metadata, placed in the slot, and recorded in a redo log file. The file system instance lands the data on the shared disk array through the redo log file; a new data block, namely a second target data block, is requested from the target shared memory and recorded in a redo log file; the metadata to be updated is written to the second target data block and recorded in a redo log file; at this point, the update processing of the metadata to be updated is completed.
[0117] 2 , there is shown a flowchart of the steps of a data management method provided by an embodiment of the present application;
[0118] A data management method, the method involving a storage end and a client end, wherein the storage end is used to process metadata, the processing including update processing and read and write processing, the method comprising:
[0119] S210: When detecting that the database instance is started, the client generates a connection request and sends the connection request to the storage end;
[0120] S220: When the client receives the key value sent by the storage end, the client connects to the shared memory according to the key value;
[0121] S230: When the client has a task to be updated, the client generates a metadata update request according to the metadata to be updated in the task to be updated, and sends the metadata update request to the storage end;
[0122] S240: When the client has a business data read / write request, the client reads the target file according to the business data read / write request and the target metadata in the shared memory.
[0123] It's important to note that the client includes the database and the database instance. The database instance is the application program that accesses the database and is a layer of data management software located between the user and the operating system. In MySQL, this is the MySQL process. Users cannot add, delete, query, modify, or perform any other operations on database files (data) directly through the file system, including data definition, data query, data maintenance, and database operation control. This requires the assistance of a database instance. These operations are performed within the database instance, which serves as the channel through which users (programmers) access the database. Applications can only interact with the database through the database instance. The database instance, as an application, encapsulates database operations and parses SQL, allowing users to manipulate database content in a simple and direct manner. Generally speaking, one database instance corresponds to one database.
[0124] It's important to note that the database manages data files in a coarse-grained manner and doesn't frequently create or resize files, requiring metadata updates. However, it places high demands on file reads and writes. When reading and writing files, it directly queries metadata through shared memory to determine the file's location on the shared disk array and directly reads and writes to the shared disk array, improving read and write performance.
[0125] As an example, the storage end updates the metadata according to the received metadata update request; since the metadata is stored in the shared memory, the client can also see the updated metadata at this time.
[0126] In a specific implementation, when the database instance is started, it communicates with the storage end of the file system to obtain the key value of the shared memory in the storage end, and connects to the shared memory after receiving the key value.
[0127] In one embodiment of the present invention, the step of reading the target file according to the business data read / write request and the target metadata in the shared memory includes:
[0128] The client determines the target metadata according to the business data read and write request;
[0129] The client determines the target disk location according to the target metadata;
[0130] The client obtains the target file according to the target disk location.
[0131] It should be noted that when the database reads and writes a file, it directly calculates the actual storage location of the file in the target shared disk through the metadata information in the shared memory, and then directly performs the read and write operations; wherein, the file size cannot be updated at the same time when writing a file; in a specific embodiment, when the database system reads and writes, it queries the metadata in the shared memory through the file system storage end to determine the location of the target file on the target shared disk, and then directly reads and writes.
[0132] In a specific implementation, the client first reads all metadata. When reading all metadata, it checks whether the data blocks corresponding to all metadata are in the existing shared memory; if not, it queries the shared memory manager whether there is new shared memory that needs to be loaded; if so, it loads the shared memory according to the specific slot information and target key value of the shared memory manager, that is, generates new shared memory; determines the third target data block from the new shared memory; and reads metadata information, that is, target metadata, from the third target data block; determines the location of the target metadata in the target shared side, that is, the target disk location, through the target metadata, and obtains the target file through the target disk location.
[0133] In one embodiment of the present invention, the step of determining the target metadata according to the business data read / write request includes:
[0134] The client obtains all metadata from the shared memory according to the business data read and write request;
[0135] The client determines all data blocks corresponding to all metadata according to all metadata;
[0136] When all the data blocks do not exist in the existing shared memory, the client queries the target manager information of the existing shared memory, and determines whether there is a new shared memory to be loaded according to the target manager information;
[0137] When the new shared memory needs to be loaded, the client obtains the slot information and the target key value in the target manager information to generate the new shared memory;
[0138] The client determines a third target data block according to the new shared memory;
[0139] The client determines the target metadata according to the third target data block.
[0140] As an example, when all the data blocks exist in the existing shared memory, the target metadata may be directly read from all the data blocks.
[0141] As an example, when there is no new shared memory to be loaded, an error message is prompted and the process ends.
[0142] In a specific implementation, all metadata in the shared memory are obtained through business data read and write requests, all data blocks corresponding to all metadata are determined in all metadata, and it is judged whether all data blocks are in the existing shared memory. When all data blocks are not in the existing shared memory, the target manager information of the existing shared memory is queried, and it is determined whether there is new shared memory that needs to be loaded through the target manager information; when there is new shared memory that needs to be loaded, the slot information and target key value in the target manager information are obtained to generate a new shared memory, and a third target data block is determined through the new shared memory; and the target metadata is determined through the third target data block.
[0143] In one embodiment of the present invention, the present application proposes a data management method to improve file system performance and reduce its overhead. Specific measures include:
[0144] 1. Use large file partitions. Unlike general file systems, databases do not have a large number of small files. Using large file partitions (this technical solution uses 8M large partitions) can greatly reduce metadata space usage. This results in a larger file allocation granularity to a certain extent, but this is completely acceptable for applications such as databases.
[0145] Second, file system metadata resides in memory. When the file system metadata space is small, it becomes possible to keep it in memory. This makes it impossible for file system metadata read operations to encounter cache misses.
[0146] 3. Directly connect to shared memory to access metadata. The file system of this technical solution is a user-mode file system, which allocates shared memory to store metadata. The most frequent operations of the database are reading and writing data files and log files. These operations are performed directly through the shared memory, minimizing overhead.
[0147] Fourth, metadata modifications are synchronously replicated. Metadata changes are recorded as logs. Metadata modifications on one node are not successful until all other nodes have received the logs and applied them to memory. Therefore, operations to add, delete, or modify directories or files must be effective on all nodes in the cluster when they are successfully returned to the storage end.
[0148] When modifying user data, the database itself will have relevant concurrency control to avoid write corruption, and the file system does not need additional concurrency control.
[0149] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0150] 3 , there is shown a structural block diagram of a data management device provided by an embodiment of the present application;
[0151] A data management device, the device involving a storage end and a client end, wherein the storage end is used to process metadata, the processing including update processing and read and write processing, and the device comprising:
[0152] A first sending module 310 is configured to receive, at the storage end, a connection request sent by the client, determine a key value corresponding to a shared memory according to the connection request, and send the key value to the client, wherein the shared memory is generated by initialization processing at the storage end;
[0153] The first allocation module 320 is used to determine the metadata to be updated based on the metadata update request sent by the client when the storage end receives the metadata update request, and allocate the metadata to be updated to the shared memory, wherein the metadata to be updated includes directories, files and file partition tables.
[0154] In one embodiment of the present invention, the first sending module 310 includes:
[0155] The first acquisition submodule is configured to determine a target shared disk according to preset configuration information.
[0156] A first processing submodule is configured to, when detecting that the target shared disk is not formatted, format the target shared disk according to the preset configuration information by the storage end, and initialize the target shared disk to obtain BOOT information;
[0157] A first determining submodule, configured for the storage end to determine quantity information according to the BOOT information;
[0158] A first construction submodule is configured to construct, on the storage end, a corresponding number of to-be-initialized shared memories according to the number of the number information, wherein at least one of the to-be-initialized shared memories is used to store manager information;
[0159] A second determining submodule is configured for the storage end to determine a disk location corresponding to each shared memory to be initialized according to the manager information;
[0160] The marking submodule is used for the storage end to add initial metadata to the corresponding shared memory to be initialized in the target disk according to the disk position, and mark the initialized shared memory as the shared memory after the initial metadata is added.
[0161] In one embodiment of the present invention, the first allocation module 320 includes:
[0162] A third determining submodule, configured for the storage end to determine whether there is a first target data block corresponding to the metadata to be updated based on the metadata to be updated;
[0163] a fourth determining submodule, configured to, when the first target data block does not exist, cause the storage end to determine whether there is an unused data block in the current shared memory;
[0164] A second construction submodule is configured to generate, by the storage end, a construction shared memory request when the current shared memory does not contain the unused data block;
[0165] A third construction submodule is used for the storage end to construct a target shared memory according to the shared memory construction request;
[0166] A fourth determining submodule is configured to cause the storage end to determine, in the target shared memory, a second target data block for storing the metadata to be updated, and to write the metadata to be updated into the second target data block.
[0167] 4 , there is shown a structural block diagram of a data management device provided by an embodiment of the present application;
[0168] A data management method, the method involving a storage end and a client end, wherein the storage end is used to process metadata, the processing including update processing and read and write processing, the method comprising:
[0169] A second sending module 410 is configured to generate a connection request by the client when detecting that the database instance is started, and send the connection request to the storage end;
[0170] A first connection module 420 is configured to connect the client to the shared memory according to the key value when the client receives the key value sent by the storage end;
[0171] The third sending module 430 is configured to, when the client has a task to be updated, generate a metadata update request based on the metadata to be updated in the task to be updated, and send the metadata update request to the storage end;
[0172] The first reading module 440 is configured to, when the client has a business data read / write request, cause the client to read a target file according to the business data read / write request and the target metadata in the shared memory.
[0173] In one embodiment of the present invention, the first reading module 440 includes:
[0174] A fifth determining submodule, configured for the client to determine the target metadata according to the business data read and write request;
[0175] A sixth determining submodule, configured for the client to determine a target disk location based on the target metadata;
[0176] The second acquisition submodule is used for the client to acquire the target file according to the target disk location.
[0177] In one embodiment of the present invention, the fifth determining submodule includes:
[0178] An acquiring unit, configured for the client to acquire all metadata from the shared memory according to the service data read and write request;
[0179] A first determining unit, configured for the client to determine all data blocks corresponding to all metadata based on all metadata;
[0180] a second determining unit, configured to, when all the data blocks do not exist in the existing shared memory, query the target manager information of the existing shared memory by the client, and determine whether there is a new shared memory to be loaded based on the target manager information;
[0181] a generating unit, configured to, when the new shared memory needs to be loaded, obtain the slot information and the target key value in the target manager information by the client to generate the new shared memory;
[0182] a third determining unit, configured for the client to determine a third target data block according to the new shared memory;
[0183] A fourth determining unit is configured to enable the client to determine the target metadata according to the third target data block.
[0184] 5 , a computer device of a data management method of the present invention is shown, which may specifically include the following:
[0185] The computer device 12 is a general-purpose computing device. The components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).
[0186] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or memory controller, a peripheral bus 18, an accelerated graphics port, a processor, or a local bus 18 that utilizes any of a variety of bus 18 architectures. Examples of such architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus 18, a Micro Channel Architecture (MAC) bus 18, an Enhanced ISA bus 18, an Audio Video Electronics Standards Association (VESA) local bus 18, and a Peripheral Component Interconnect (PCI) bus 18.
[0187] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.
[0188] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although not shown in FIG. 5 , a magnetic disk drive may be provided for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), as well as an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media). In these cases, each drive may be connected to bus 18 via one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42 configured to perform the functions of various embodiments of the present invention.
[0189] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in a memory. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules 42, and program data, each of which, or some combination thereof, may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0190] The computer device 12 can also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, camera, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication can occur via an input / output (I / O) interface 22. Furthermore, the computer device 12 can communicate with one or more networks (e.g., a local area network (LAN)), a wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in FIG. 5 , other hardware and / or software modules can be used in conjunction with the computer device 12, including but not limited to microcode, device drivers, redundant processing units 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34.
[0191] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a data management method provided by an embodiment of the present invention.
[0192] That is, when the processing unit 16 executes the program, the following is achieved: the storage end receives the connection request sent by the client, determines the key value corresponding to the shared memory based on the connection request, and sends the key value to the client, wherein the shared memory is generated by the initialization processing of the storage end; when the storage end receives the metadata update request sent by the client, the storage end determines the metadata to be updated based on the metadata update request, and allocates the metadata to be updated to the shared memory, wherein the metadata to be updated includes directories, files, and file partition tables.
[0193] In an embodiment of the present invention, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a data management method as provided in all embodiments of the present application:
[0194] That is, when the program is executed by the processor, it is implemented as follows: the storage end receives a connection request sent by the client, determines a key value corresponding to the shared memory based on the connection request, and sends the key value to the client, wherein the shared memory is generated by initialization processing by the storage end; when the storage end receives a metadata update request sent by the client, the storage end determines metadata to be updated based on the metadata update request, and allocates the metadata to be updated to the shared memory, wherein the metadata to be updated includes directories, files, and file partition tables.
[0195] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0196] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0197] The computer program code for performing the operations of the present invention can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.
[0198] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0199] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.
[0200] The above is a detailed introduction to a data management method, device, equipment and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A data management method, It is characterized in that The method involves a storage end and a client end, wherein the storage end is used for processing metadata, the processing including update processing and read-write processing, and the method includes: The storage end receives a connection request sent by the client, determines a key value corresponding to a shared memory according to the connection request, and sends the key value to the client, wherein the shared memory is generated by initialization processing of the storage end; When the storage end receives the metadata update request sent by the client, the storage end determines the metadata to be updated according to the metadata update request, and allocates and processes the metadata to be updated to the shared memory, wherein the metadata to be updated includes directories, files, and file partition tables.
2. The method according to claim 1, It is characterized in that The step of generating the shared memory by initialization processing of the storage end includes: Determine the target shared disk based on the preset configuration information. When it is detected that the target shared disk is not formatted, the storage end formats the target shared disk according to the preset configuration information, and initializes the target shared disk to obtain BOOT information; The storage end determines the number information according to the BOOT information; The storage end constructs a corresponding number of to-be-initialized shared memories according to the number of the number information, wherein at least one of the to-be-initialized shared memories is used to store the manager information; The storage end determines the disk location corresponding to each shared memory to be initialized according to the manager information; The storage end adds the initial metadata to the corresponding shared memory to be initialized in the target disk according to the disk position, and marks the initialized shared memory as the shared memory after the initial metadata is added.
3. The method according to claim 1, It is characterized in that The step of determining the metadata to be updated according to the metadata update request, and allocating the metadata to be updated to the shared memory, wherein the metadata to be updated includes directories, files, and file partition tables, comprises: The storage end determines whether there is a first target data block corresponding to the metadata to be updated according to the metadata to be updated; When the first target data block does not exist, the storage end determines whether there is an unused data block in the current shared memory; When the current shared memory does not contain the unused data block, the storage end generates a request to construct a shared memory; The storage end constructs a target shared memory according to the shared memory construction request; The storage end determines a second target data block in the target shared memory for storing the metadata to be updated, and writes the metadata to be updated into the second target data block.
4. A data management method, It is characterized in that The method involves a storage end and a client end, wherein the storage end is used for processing metadata, the processing including update processing and read-write processing, and the method includes: When detecting that the database instance is started, the client generates a connection request and sends the connection request to the storage end; When the client receives the key value sent by the storage end, the client connects to the shared memory according to the key value; When the client has a task to be updated, the client generates a metadata update request according to the metadata to be updated in the task to be updated, and sends the metadata update request to the storage end; When the client has a service data read / write request, the client reads the target file according to the service data read / write request and the target metadata in the shared memory.
5. The method according to claim 4, It is characterized in that The step of reading the target file according to the business data read / write request and the target metadata in the shared memory includes: The client determines the target metadata according to the business data read and write request; The client determines the target disk location according to the target metadata; The client obtains the target file according to the target disk location.
6. The method according to claim 5, It is characterized in that The step of determining the target metadata according to the business data read / write request comprises: The client obtains all metadata in the shared memory according to the business data read and write request; The client determines all data blocks corresponding to all metadata according to all metadata; When all the data blocks are not in the existing shared memory, the client queries the target manager information of the existing shared memory, and determines whether there is a new shared memory to be loaded through the target manager information; When the new shared memory needs to be loaded, the client obtains the slot information and the target key value in the target manager information to generate the new shared memory; The client determines a third target data block according to the new shared memory; The client determines the target metadata according to the third target data block.
7. A data management device, It is characterized in that The device involves a storage end and a client end, wherein the storage end is used for processing metadata, the processing including update processing and read-write processing, and the device includes: A first sending module, configured for the storage end to receive a connection request sent by the client, and to determine a key value corresponding to a shared memory according to the connection request, and to send the key value to the client, wherein the shared memory is generated by initialization processing of the storage end; The first allocation module is used to determine the metadata to be updated according to the metadata update request when the storage end receives the metadata update request sent by the client, and allocate the metadata to be updated to the shared memory, wherein the metadata to be updated includes directories, files and file partition tables.
8. A data management device, It is characterized in that The device involves a storage end and a client end, wherein the storage end is used for processing metadata, the processing including update processing and read-write processing, and the device includes: A second sending module, configured to, when detecting that a database instance is started, cause the client to generate a connection request and send the connection request to the storage end; A first connection module, configured to connect the client to the shared memory according to the key value when the client receives the key value sent by the storage end; A third sending module, configured to, when there is a task to be updated on the client, cause the client to generate a metadata update request according to the metadata to be updated in the task to be updated, and send the metadata update request to the storage end; The first reading module is used for, when the client has a business data read / write request, the client reads the target file according to the business data read / write request and the target metadata in the shared memory.
9. A computer device, It is characterized in that The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 6 when executed by the processor.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Direct access to persistent memory of shared storage
CN105408880A
RDMA (Remote Direct Memory Access)-based distributed memory file system
CN108268208A
Data processing method and device
CN109889561A
RDMA-based key value storage system and working method thereof
CN110147345A
Data operation method and system
CN116028677A
