Data management method applied to distributed storage system and related equipment

Abstract

The embodiment of the invention belongs to the technical field of data processing and the field of medical treatment and health, and relates to a data management method applied to a distributed storage system and related equipment.The method comprises the steps that the storage system is constructed according to a Bloom filter and a consistent Hash algorithm, and when data needs to be updated, the Bloom filter and the consistent Hash algorithm are updated; and the storage layer is responsible for changing a stored database table application and performing distributed execution on a redo log. The Bloom filter and the consistency Hash algorithm are introduced, so that the storage space and the processing speed are optimized, and the consistency and the reliability of the data are enhanced through the version vector and a redo log mechanism.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, is applicable to the field of medical health, and in particular, to a data management method and related equipment applied to a distributed storage system. Background Art

[0002] As the complexity of data management systems increases, the application of distributed database technology is becoming more and more widespread, such as medical platforms that support disease-assisted diagnosis, health management, remote consultation and other functions.

[0003] While most existing relational databases are not distributed, some existing databases are "scaled" (rather than just "scaling" by using a larger monolithic system) using one of two common models: a "shared nothing" model and a "shared disk" model.

[0004] In the "non-shared" model, incoming queries are broken into database shards (each containing a component of the query), which are sent to different compute nodes for query processing, and the results are collected and aggregated before being returned. Typically, in the "shared disk" model, every compute node in the cluster has access to the same underlying data. Cache coherency must be managed very carefully in systems that adopt this model. In both models, a large monolithic database (including all the functionality of independent database instances) is replicated on multiple nodes, and "glue" logic is added to stitch them together.

[0005] In the "shared disk" model, glue logic can be used to fuse the caches of multiple nodes together (e.g., managing the consistency of the cache layer). These "non-shared" and "shared disk" database systems are expensive to deploy, complex to maintain, and may over-serve many database use cases. In traditional database systems, the data managed by the database system is stored on direct-attached disks. If a disk fails, it is replaced and the corresponding data must then be reloaded.

[0006] However, the applicant has found that traditional databases cannot be well extended to large databases. Summary of the invention

[0007] The purpose of the embodiments of the present application is to provide a data management method and related devices applied to a distributed storage system to solve the problem that traditional databases cannot be well expanded to large databases.

[0008] In order to solve the above technical problems, the present application embodiment provides a data management method applied to a distributed storage system, which adopts the following technical solution:

[0009] Build storage layer metadata index based on database logical structure;

[0010] Using a Bloom filter to compress and store the metadata index to obtain a compressed metadata index;

[0011] The compressed metadata index is stored in shards according to a consistent hashing algorithm, wherein the compressed metadata index also includes version vector data;

[0012] When receiving a data change request sent by a user terminal, obtaining an index of the data to be changed and a data change strategy according to the data change request;

[0013] Extracting the data to be changed corresponding to the index of the data to be changed, and updating the version vector data of the data to be changed according to the data change strategy to obtain an updated version vector;

[0014] Performing a data update operation on the data to be changed according to the data change strategy, and redoing the log record according to the data update operation and the update version vector to obtain a redo log record;

[0015] The redo log records are executed in a distributed manner.

[0016] Further, comparing the version vectors of the local database and the received redo log to obtain a version vector comparison result;

[0017] If the version vector comparison result does not require concurrent modification, the redo log record is executed in a distributed manner;

[0018] If the version vector comparison result needs to be modified concurrently, a data merge operation is performed according to a preset conflict resolution strategy.

[0019] Further, when the distributed storage system crashes and needs to be restored;

[0020] The background threads of the distributed storage system operate in parallel on different storage nodes to perform data page reconstruction processing according to the redo log.

[0021] Furthermore, the step of storing the compressed metadata in fragments according to the consistent hashing algorithm specifically includes the following steps:

[0022] Performing hash mapping on the metadata according to the MurmurHash hash function to obtain a hash value;

[0023] Obtaining the number of virtual nodes corresponding to each physical node according to the hash value, and performing allocation processing on the number of virtual nodes to obtain virtual node setting data;

[0024] According to the virtual node setting data, the metadata is segmented by keyword using a consistent hashing algorithm to obtain data segments stored on each virtual node;

[0025] Through the data sharding strategy, a multi-copy mechanism is adopted to perform redundant storage for each data shard.

[0026] Furthermore, the step of storing the compressed metadata index in shards according to the consistent hashing algorithm specifically includes the following steps:

[0027] Mapping the metadata to points in a hash space;

[0028] Divide the hash space into a plurality of intervals, wherein each of the intervals corresponds to a shard node;

[0029] The hash value of each metadata index is calculated according to the hash function, and the metadata index is mapped to the corresponding shard node according to the interval where the hash value is located.

[0030] Furthermore, the step of distributing the redo log records specifically includes the following steps:

[0031] Synchronizing the redo log to distributed nodes through a distributed message queue;

[0032] After the distributed node receives the redo log, it updates the local data according to the data change information recorded in the log.

[0033] In order to solve the above technical problems, the embodiment of the present application further provides a data management device applied to a distributed storage system, which adopts the following technical solution:

[0034] Index building module, used to build storage layer metadata index according to database logical structure;

[0035] A compression storage module, used for compressing and storing the metadata index using a Bloom filter to obtain a compressed metadata index;

[0036] A shard storage module, used for sharding and storing the compressed metadata index according to a consistent hashing algorithm, wherein the compressed metadata index also includes version vector data;

[0037] A request acquisition module, configured to, when receiving a data change request sent by a user terminal, acquire an index of the data to be changed and a data change strategy according to the data change request;

[0038] A vector update module, used for extracting the data to be changed corresponding to the index of the data to be changed, and updating the version vector data of the data to be changed according to the data change strategy to obtain an updated version vector;

[0039] A log redo module, configured to perform a data update operation on the data to be changed according to the data change strategy, and redo log records according to the data update operation and the update version vector to obtain redo log records;

[0040] The distributed execution module is used to execute the redo log records in a distributed manner.

[0041] Furthermore, the distributed execution module includes:

[0042] A vector version comparison submodule, used to compare the version vectors of the local database and the received redo log to obtain a version vector comparison result;

[0043] A data update submodule, configured to perform a data update operation according to the redo log if the version vector comparison result does not require concurrent modification;

[0044] The data merging submodule is used to perform a data merging operation according to a preset conflict resolution strategy if the version vector comparison result needs to be modified concurrently.

[0045] Furthermore, the device also includes:

[0046] A crash recovery module is used for when the distributed storage system crashes and needs to be recovered, the background threads of the distributed storage system operate in parallel on different storage nodes to perform data page reconstruction processing according to the redo log.

[0047] Furthermore, the shard storage module includes:

[0048] A hash mapping submodule, used for performing hash mapping on the metadata according to the MurmurHash hash function to obtain a hash value;

[0049] An allocation submodule, used for obtaining the number of virtual nodes corresponding to each physical node according to the hash value, and performing allocation processing on the number of virtual nodes to obtain virtual node setting data;

[0050] A sharding submodule is used to set data according to the virtual node, and use a consistent hashing algorithm to shard the metadata by keyword to obtain data shards stored on each virtual node;

[0051] The storage submodule is used to perform redundant storage for each data shard through the data sharding strategy and the multi-copy mechanism.

[0052] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0053] The system comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data management method applied to a distributed storage system as described above when executing the computer-readable instructions.

[0054] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0055] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data management method applied to a distributed storage system as described above are implemented.

[0056] The present application provides a data management method for a distributed storage system, including: constructing a storage layer metadata index according to a database logical structure; compressing and storing the metadata index using a Bloom filter to obtain a compressed metadata index; sharding and storing the compressed metadata index according to a consistent hashing algorithm, wherein the compressed metadata index also includes version vector data; when a data change request sent by a user terminal is received, obtaining a data index to be changed and a data change strategy according to the data change request; extracting the data to be changed corresponding to the data index to be changed, and updating the version vector data of the data to be changed according to the data change strategy to obtain an updated version vector; performing a data update operation on the data to be changed according to the data change strategy, and redoing a log record according to the data update operation and the updated version vector to obtain a redo log record; and executing the redo log record in a distributed manner. Compared with the prior art, the present application introduces a Bloom filter and a consistent hashing algorithm, so that data can be distributed as evenly as possible on each node. This even distribution effectively improves the utilization of storage space, thereby optimizing storage space and processing speed. In addition, the consistency and reliability of data are enhanced through the version vector and redo log mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0059] Figure 2 is a flowchart of a data management method applied to a distributed storage system provided in Embodiment 1 of the present application;

[0060] Figure 3 is a structural diagram of a data management device applied to a distributed storage system provided in Embodiment 2 of the present application;

[0061] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0063] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0064] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0065] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0066] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0067] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0068] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0069] It should be noted that the data management method applied to the distributed storage system provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the data management device applied to the distributed storage system is generally arranged in the server / terminal device.

[0070] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0071] Embodiment 1

[0072] Continue to refer Figure 2 , shows a flow chart of an embodiment of a data management method applied to a distributed storage system according to the present application. The data management method applied to a distributed storage system comprises: step S201, step S202, step S203, step S204, step S205, step S206 and step S207.

[0073] In step S201, a storage layer metadata index is constructed according to the database logical structure.

[0074] In step S202, a Bloom filter is used to compress and store the metadata index to obtain a compressed metadata index.

[0075] In step S203, the compressed metadata index is stored in shards according to a consistent hashing algorithm, wherein the compressed metadata index also includes version vector data.

[0076] In step S204, when a data modification request sent by the user terminal is received, the index of the data to be modified and the data modification strategy are obtained according to the data modification request.

[0077] In step S205, the data to be changed corresponding to the index of the data to be changed is extracted, and the version vector data of the data to be changed is updated according to the data change strategy to obtain an updated version vector.

[0078] In step S206, a data update operation is performed on the data to be changed according to the data change policy, and a redo log record is redone according to the data update operation and the update version vector to obtain a redo log record.

[0079] In step S207, the redo log records are executed in a distributed manner.

[0080] In an embodiment of the present application, some of the lowest level operations of the database (e.g., backup, recovery, snapshot, recovery, and various space management operations) can be delegated from the database engine to the storage layer and distributed across multiple nodes and storage devices. For example, rather than the database engine applying changes to a database table (or its data page) and then sending the modified data page to the storage layer, the storage layer itself is responsible for applying changes to the stored database table (and its data page). In such use, redo log records instead of modified data pages can be sent to the storage layer, after which the redo process (e.g., application of redo log records) can be slightly delayed and executed in a distributed manner (e.g., by background processing). Crash recovery (e.g., rebuilding data pages from stored redo log records) can also be performed by the storage layer, and can also be performed by distributed background processing.

[0081] In the embodiment of the present application, the database is mainly used to store medical data, such as personal health records, prescriptions, examination reports, medical images and other data, wherein medical images refer to internal tissues obtained in a non-invasive manner for medical treatment or medical research, for example, images of the stomach, abdomen, heart, knees, and brain, such as CT (Computed Tomogr aphy), MRI (Magnetic Resonance Imaging), US (ultrasound), X-ray images, electroencephalograms, and optical photographic lights generated by medical instruments.

[0082] In the embodiment of the present application, because only redo logs (rather than modified data pages) are sent to the storage layer, the network traffic between the database layer and the storage layer may be much less than in existing database systems. In some uses, the size of each redo log may be one-tenth the size of the corresponding data page to which it specifies the change. The requests sent from the database layer and the distributed storage system may be asynchronous, and there may be multiple such requests at a time.

[0083] In an embodiment of the present application, a data management method for a distributed storage system is provided, including: constructing a storage layer metadata index according to a database logical structure; compressing and storing the metadata index using a Bloom filter to obtain a compressed metadata index; sharding and storing the compressed metadata index according to a consistent hashing algorithm, wherein the compressed metadata index also includes version vector data; when a data change request sent by a user terminal is received, obtaining a data index to be changed and a data change strategy according to the data change request; extracting the data to be changed corresponding to the data index to be changed, and updating the version vector data of the data to be changed according to the data change strategy to obtain an updated version vector; performing a data update operation on the data to be changed according to the data change strategy, and redoing the log record according to the data update operation and the updated version vector to obtain a redo log record; and executing the redo log record in a distributed manner. Compared with the prior art, the present application introduces a Bloom filter and a consistent hashing algorithm, so that data can be distributed as evenly as possible on each node. This even distribution effectively improves the utilization of storage space, thereby optimizing storage space and processing speed. In addition, the consistency and reliability of data are enhanced through the version vector and redo log mechanism.

[0084] In some optional implementations of the embodiments of the present application, the above step S207 specifically includes the following steps:

[0085] Compare the version vectors of the local database and the received redo log to obtain the version vector comparison result;

[0086] If the version vector comparison result does not require concurrent modification, the data update operation is performed according to the redo log;

[0087] If the version vector comparison result needs to be modified concurrently, the data merge operation is performed according to the preset conflict resolution strategy.

[0088] In some optional implementations of the embodiments of the present application, after step S207, the following steps are further included:

[0089] When the distributed storage system crashes and needs to be recovered;

[0090] The background threads of the distributed storage system operate in parallel on different storage nodes to perform data page reconstruction processing based on the redo logs.

[0091] In an embodiment of the present application, a database system can restart a database engine almost immediately after a database crash (e.g., to make the database available to accept and service queries) without having to wait for the entire data set to be recovered. Instead, queries can be executed while one or more background threads delay performing crash recovery. For example, after a crash, multiple background threads can operate in parallel on different storage nodes to rebuild data pages from corresponding redo logs. At the same time, if an incoming query targets a data page that has not yet been reconstructed, the storage layer can be configured to dynamically recreate the data page from the appropriate redo log.

[0092] In some optional implementations of the embodiments of the present application, step S203 specifically includes the following steps:

[0093] Perform hash mapping on metadata according to MurmurHash hash function to obtain a hash value;

[0094] The number of virtual nodes corresponding to each physical node is obtained according to the hash value, and the number of virtual nodes is allocated and processed to obtain virtual node setting data;

[0095] According to the virtual node setting data, the metadata is sharded by keyword using the consistent hashing algorithm to obtain the data shards stored on each virtual node;

[0096] Through the data sharding strategy, a multi-copy mechanism is adopted to perform redundant storage for each data shard.

[0097] In the embodiment of the present application, when designing an efficient and robust distributed storage system, we first use the MurmurHash hash function to hash the key value of the metadata. This algorithm is highly favored for its high efficiency and hash value with low collision rate. Then, according to the obtained hash value, we use the consistent hashing algorithm to allocate virtual nodes.

[0098] In the embodiment of the present application, in order to improve the reliability of data, each data shard will be stored on at least three different physical nodes, forming a multi-copy storage mechanism for data, so that when any node fails, the copies on other nodes can be used to restore the data, thereby ensuring high availability of data.

[0099] In an embodiment of the present application, through the distributed coordination service ZooKeeper, the system can monitor the data update status of each node and synchronize the data updates of all replicas to ensure global data consistency.

[0100] In an embodiment of the present application, as the system runs, the addition and removal of nodes will trigger data redistribution in the consistent hashing algorithm.

[0101] In the embodiment of the present application, in order to minimize the data migration caused by node changes, we adopt the minimum migration principle and only migrate the minimum affected data set, which not only reduces the network transmission load, but also optimizes the overall performance of the system.

[0102] In the embodiment of the present application, in addition, in order to improve data retrieval efficiency, the system will also establish a secondary index and cache mechanism for high-frequency access data.

[0103] In an embodiment of the present application, by analyzing data access patterns, hot data such as user access records and frequently queried metadata indexes are cached in memory, thereby greatly improving data access speed.

[0104] In the embodiments of the present application, an efficient and reliable distributed metadata management system can be designed to ensure uniform distribution, high availability and fast retrieval of data, thereby meeting the stringent requirements in a large-scale distributed storage environment.

[0105] In some optional implementations of the embodiments of the present application, the above step S203 specifically includes the following steps:

[0106] Map metadata to points in hash space;

[0107] Divide the hash space into several intervals, where each interval corresponds to a shard node;

[0108] The hash value of each metadata index is calculated according to the hash function, and the metadata index is mapped to the corresponding shard node according to the interval of the hash value.

[0109] In an embodiment of the present application, in order to achieve efficient indexing and query of metadata, a consistent hashing algorithm can be used for sharded storage. Specifically, first, according to the characteristics of the metadata, such as file name, creation time, size, etc., a suitable hash function is designed, such as the MurmurHash3 algorithm, to map the metadata to a hash space of size 2^32; then, the hash space is divided into 1024 intervals, each interval corresponding to a shard node; for each metadata index, its hash value is calculated by MurmurHash3, and according to the interval where the hash value is located, the index is mapped to the corresponding shard node.

[0110] In an embodiment of the present application, an inverted index structure can be used on a shard node to establish a mapping from keywords to document IDs, thereby supporting fast full-text retrieval.

[0111] In an embodiment of the present application, in order to ensure load balancing, virtual node technology can be introduced to map each shard node to 128 virtual nodes, increase the discreteness of the hash space, and allow metadata to be more evenly distributed on each node.

[0112] In an embodiment of the present application, during the index writing and query process, through the characteristics of the consistent hashing algorithm, even when the number of nodes increases or decreases, only an average of 1 / 1024 of the data needs to be migrated, thereby ensuring the high availability and stability of the system.

[0113] In an embodiment of the present application, for hot indexes, such as certain popular keywords, an LRU cache strategy can be used to cache the 10,000 most recently accessed index data in memory, thereby improving the query response speed and reducing the I / O pressure on the sharding node.

[0114] In the embodiments of the present application, an efficient, scalable, load-balanced metadata indexing and query system can be implemented.

[0115] In some optional implementations of the embodiments of the present application, the above step S207 specifically includes the following steps:

[0116] The redo log is synchronized to the distributed nodes through the distributed message queue;

[0117] After the distributed node receives the redo log, it updates the local data according to the data change information recorded in the log.

[0118] In an embodiment of the present application, the system writes the generated redo log into a distributed message queue such as Kafka, and pushes the log message to other nodes asynchronously through a topic subscription mechanism to ensure that the redo log sequence numbers of each node are continuous and consistent.

[0119] In an embodiment of the present application, each node subscribes to a message queue, receives redo logs in real time, parses the log content, locates local data based on information such as table name and primary key, updates the data through SQL statements, and upon completion, persists the updated data and the latest version vector to the local MySQL database.

[0120] In an embodiment of the present application, the system uses the Paxos algorithm to vote among the nodes every 10 minutes, and obtains the version vector recognized by more than half of the nodes as the global data version. Through version number comparison and incremental synchronization, the final consistency of the data is finally achieved.

[0121] In an embodiment of the present application, the system samples and analyzes the generation rate and occupied space of the redo log every 1 minute. When the log rate exceeds 1000 logs / second or the occupied space exceeds 1GB, the synchronization strategy is adjusted, and optimization methods such as batch merging and asynchronous disk flushing are used to maximize system performance without affecting data consistency.

[0122] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0123] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0125] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0126] Embodiment 2

[0127] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a data management device for a distributed storage system. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0128] like Figure 3 As shown, the data management device 200 applied to the distributed storage system in the embodiment of the present application includes:

[0129] An index building module 210 is used to build a storage layer metadata index according to the database logical structure;

[0130] A compression storage module 220 is used to compress and store the metadata index using a Bloom filter to obtain a compressed metadata index;

[0131] A shard storage module 230, configured to shard the compressed metadata index according to a consistent hashing algorithm, wherein the compressed metadata index also includes version vector data;

[0132] The request acquisition module 240 is used to acquire the index of the data to be changed and the data change strategy according to the data change request when receiving the data change request sent by the user terminal;

[0133] The vector update module 250 is used to extract the data to be changed corresponding to the index of the data to be changed, and update the version vector data of the data to be changed according to the data change strategy to obtain an updated version vector;

[0134] A log redo module 260 is used to perform a data update operation on the data to be changed according to the data change strategy, and redo the log record according to the data update operation and the update version vector to obtain a redo log record;

[0135] The distributed execution module 270 is used to execute the redo log records in a distributed manner.

[0136] In an embodiment of the present application, a data management device 200 applied to a distributed storage system is provided, including: an index construction module 210, which is used to construct a storage layer metadata index according to a database logical structure; a compression storage module 220, which is used to compress and store the metadata index using a Bloom filter to obtain a compressed metadata index; a shard storage module 230, which is used to shard and store the compressed metadata index according to a consistent hashing algorithm, wherein the compressed metadata index also includes version vector data; a request acquisition module 240, which is used to obtain a to-be-changed data index and a data change strategy according to the data change request when a data change request sent by a user terminal is received; a vector update module 250, which is used to extract the to-be-changed data corresponding to the to-be-changed data index, and update the version vector data of the to-be-changed data according to the data change strategy to obtain an updated version vector; a log redo module 260, which is used to perform a data update operation on the to-be-changed data according to the data change strategy, and redo log records according to the data update operation and the updated version vector to obtain a redo log record; and a distributed execution module 270, which is used to execute the redo log record in a distributed manner. Compared with the prior art, this application introduces Bloom filters and consistent hashing algorithms, so that data can be distributed on each node as evenly as possible. This uniform distribution effectively improves the utilization of storage space, thereby optimizing storage space and processing speed. In addition, the consistency and reliability of data are enhanced through version vectors and redo log mechanisms.

[0137] In some optional implementations of the embodiments of the present application, the above-mentioned distributed execution module includes:

[0138] The vector version comparison submodule is used to compare the version vectors of the local database and the received redo log to obtain the version vector comparison result;

[0139] The data update submodule is used to perform data update operations according to the redo log if the version vector comparison result does not require concurrent modification;

[0140] The data merging submodule is used to perform data merging operations according to a preset conflict resolution strategy if the version vector comparison result needs to be modified concurrently.

[0141] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0142] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0143] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0144] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions for a data management method applied to a distributed storage system. In addition, the memory 310 can also be used to temporarily store various types of data that have been output or are to be output.

[0145] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions stored in the memory 310 or process data, such as running the computer-readable instructions of the data management method applied to the distributed storage system.

[0146] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0147] The computer device provided in this application introduces a Bloom filter and a consistent hashing algorithm so that data can be distributed as evenly as possible on each node. This uniform distribution effectively improves the utilization of storage space, thereby optimizing storage space and processing speed. In addition, the consistency and reliability of data are enhanced through version vectors and redo log mechanisms.

[0148] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data management method applied to the distributed storage system as described above.

[0149] The computer-readable storage medium provided in the present application introduces a Bloom filter and a consistent hashing algorithm, so that data can be distributed on each node as evenly as possible. This uniform distribution effectively improves the utilization of storage space, thereby optimizing storage space and processing speed. In addition, the consistency and reliability of data are enhanced through version vectors and redo log mechanisms.

[0150] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0151] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data management method applied to a distributed storage system, characterized in that: The steps include: Build storage layer metadata index based on database logical structure; Using a Bloom filter to compress and store the metadata index to obtain a compressed metadata index; The compressed metadata index is stored in shards according to a consistent hashing algorithm, wherein the compressed metadata index also includes version vector data; When receiving a data change request sent by a user terminal, obtaining an index of the data to be changed and a data change strategy according to the data change request; Extracting the data to be changed corresponding to the index of the data to be changed, and updating the version vector data of the data to be changed according to the data change strategy to obtain an updated version vector; Performing a data update operation on the data to be changed according to the data change strategy, and redoing the log record according to the data update operation and the update version vector to obtain a redo log record; The redo log records are executed in a distributed manner.

2. The data management method applied to a distributed storage system according to claim 1, characterized in that: The step of distributing the redo log records specifically includes the following steps: Comparing the version vectors of the local database and the received redo log to obtain a version vector comparison result; If the version vector comparison result does not require concurrent modification, the redo log record is executed in a distributed manner; If the version vector comparison result needs to be modified concurrently, a data merge operation is performed according to a preset conflict resolution strategy.

3. The data management method applied to a distributed storage system according to claim 1, characterized in that: After the step of distributing the redo log records, the following steps are also included: When the distributed storage system crashes and needs to be recovered, the background threads of the distributed storage system operate in parallel on different storage nodes to perform data page reconstruction processing according to the redo log.

4. The data management method applied to a distributed storage system according to claim 3, characterized in that: The step of storing the compressed metadata in segments according to the consistent hashing algorithm specifically includes the following steps: Performing hash mapping on the metadata according to the MurmurHash hash function to obtain a hash value; Obtaining the number of virtual nodes corresponding to each physical node according to the hash value, and performing allocation processing on the number of virtual nodes to obtain virtual node setting data; According to the virtual node setting data, the metadata is segmented by keyword using a consistent hashing algorithm to obtain data segments stored on each virtual node; Through the data sharding strategy, a multi-copy mechanism is adopted to perform redundant storage for each data shard.

5. The data management method applied to a distributed storage system according to claim 1, characterized in that: The step of storing the compressed metadata index in shards according to the consistent hashing algorithm specifically includes the following steps: Mapping the metadata to points in a hash space; Divide the hash space into a plurality of intervals, wherein each of the intervals corresponds to a shard node; The hash value of each metadata index is calculated according to the hash function, and the metadata index is mapped to the corresponding shard node according to the interval where the hash value is located.

6. The data management method applied to a distributed storage system according to claim 1, characterized in that: The step of distributing the redo log records specifically includes the following steps: Synchronizing the redo log to distributed nodes through a distributed message queue; After the distributed node receives the redo log, it updates the local data according to the data change information recorded in the log.

7. A data management device applied to a distributed storage system, characterized in that: include: Index building module, used to build storage layer metadata index according to database logical structure; A compression storage module, used to compress and store the metadata index using a Bloom filter to obtain a compressed metadata index; A shard storage module, used for sharding and storing the compressed metadata index according to a consistent hashing algorithm, wherein the compressed metadata index also includes version vector data; A request acquisition module, configured to, when receiving a data change request sent by a user terminal, acquire an index of the data to be changed and a data change strategy according to the data change request; A vector update module, used for extracting the data to be changed corresponding to the index of the data to be changed, and updating the version vector data of the data to be changed according to the data change strategy to obtain an updated version vector; A log redo module, configured to perform a data update operation on the data to be changed according to the data change strategy, and redo log records according to the data update operation and the update version vector to obtain redo log records; The distributed execution module is used to execute the redo log records in a distributed manner.

8. The data management device applied to a distributed storage system according to claim 7, characterized in that: The distributed execution module includes: A vector version comparison submodule, used to compare the version vectors of the local database and the received redo log to obtain a version vector comparison result; A data update submodule, configured to perform a data update operation according to the redo log if the version vector comparison result does not require concurrent modification; The data merging submodule is used to perform a data merging operation according to a preset conflict resolution strategy if the version vector comparison result needs to be modified concurrently.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the data management method applied to a distributed storage system according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data management method applied to a distributed storage system according to any one of claims 1 to 6 are implemented.

Images