Metadata asynchronous disk brushing method based on data stock calculation separation architecture

Abstract

The embodiment of the invention provides a metadata asynchronous disk refreshing method based on a data stock calculation separation architecture, which comprises the following steps: receiving a modification instruction of target metadata in a shared storage node, locking the target metadata according to the modification instruction, and modifying the target metadata in a shared memory of the shared storage node; constructing a data log of the target metadata operation, and writing the data log into a disk; copying the modified target metadata into a memory pool, mounting the modified target metadata into a task queue of a background metadata disk brushing thread, and unlocking the modified target metadata; performing disk refreshing operation on the modified target metadata in the memory pool through a background metadata disk refreshing thread according to a task sequence mounted in the task queue; and after the modified target metadata is brushed, data logs in the disk are emptied. According to the embodiment of the invention, the holding time of the lock in the metadata modification process is shortened, and the lock conflict time is shortened.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a metadata asynchronous disk flushing method based on a database storage and computing separation architecture. Background Art

[0002] In the related art, the database computing separation architecture is an architectural design that deploys and manages data storage and data processing functions separately. Under this architecture, storage methods can be mainly divided into distributed storage and shared storage. The advantage of shared storage is that multiple computing nodes can access the same data together, reducing redundant data storage and reducing storage costs. However, in the prior art, since the master node needs to frequently append to the file, it needs to modify the metadata first and then write the data, and the backup node frequently reads the file, it needs to read the metadata first and then read the data, resulting in frequent locking and unlocking operations and long lock waiting times, affecting the file operation speed, and thus affecting the performance of the overall storage and computing separation architecture. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a metadata asynchronous disk flushing method based on a database computing separation architecture, aiming to reduce the lock holding time in the metadata modification process and thus reduce the lock conflict time.

[0004] In a first aspect, an embodiment of the present application provides a metadata asynchronous disk flushing method based on a database and computing separation architecture, wherein the separation architecture includes a shared storage node, a background metadata disk flushing thread, and a memory pool, and the method includes: receiving a modification instruction of the target metadata in the shared storage node, locking the target metadata according to the modification instruction, and performing a modification operation on the target metadata in the shared memory of the shared storage node; Constructing a data log of the target metadata operation and writing the data log to a disk; The modified target metadata is copied to the memory pool, mounted to the task queue of the background metadata flushing thread, and the modified target metadata is unlocked; The modified target metadata in the memory pool is flushed to disk by the background metadata flushing thread according to the order of tasks mounted in the task queue; After the modified target metadata is flushed to disk, the data log in the disk is cleared.

[0005] According to some embodiments of the present application, the method further includes: Dividing the disk into a plurality of regions according to a first granularity, and numbering the plurality of regions to obtain a plurality of region numbers; Divide the background metadata disk flushing thread into multiple threads, and number the multiple threads to obtain multiple thread numbers; A target relationship table is obtained according to the plurality of area numbers and the plurality of thread numbers. According to some embodiments of the present application, in the case of obtaining the target relationship table, the method includes: An instruction for adding the target metadata is received, a thread number of the target metadata is obtained according to the target relationship table, and the target metadata is added to a preset thread according to the thread number.

[0006] According to some embodiments of the present application, obtaining the thread number of the target metadata according to the target relationship table includes: Obtaining the quantity value of the regions and the quantity value of the threads according to the target relationship table; Calculating the number value of the regions and the number value of the threads to obtain a calculated value; The thread number of the target metadata is obtained according to the calculated value.

[0007] According to some embodiments of the present application, the method further includes: Creating a chain of the target metadata and placing the target metadata into the chain to form a history chain and obtaining a first version of the target metadata; The target metadata is placed at the head of the history chain to form an intermediate chain and a second version of the target metadata is obtained, wherein the second version of the target metadata points to the first version of the target metadata.

[0008] According to some embodiments of the present application, the method further includes: When the background metadata flushing thread includes the target metadata, the target metadata is placed at the head of the intermediate chain to form a target chain and a third version of the target metadata is obtained, and the third version of the target metadata points to the second version of the target metadata.

[0009] According to some embodiments of the present application, when the first version of the target metadata, the second version of the target metadata, and the third version of the target metadata are obtained, the method further includes: After writing the data log to the disk, the first version of the target metadata or the first version of the metadata and the second version of the metadata in the disk are cleared, and the historical chain or the intermediate chain is cleared. According to some embodiments of the present application, when the arrangement order of the chain of the target metadata is the order of the sequence number of the target metadata from large to small, it includes: Each time the target metadata is modified, the value of the sequence number of the target metadata is changed.

[0010] In a second aspect, an embodiment of the present application provides a computer device, including: at least one memory; at least one processor; at least one computer program; The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement the metadata asynchronous disk flushing method based on the database and computation separation architecture described in the first aspect.

[0011] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the metadata asynchronous disk flushing method based on the database and computing separation architecture described in the first aspect above.

[0012] According to the technical solution of the embodiment of the present application, at least the following beneficial effects are achieved: in the case where the separation architecture includes a shared storage node, a background metadata disk flushing thread and a memory pool, a modification instruction of the target metadata in the shared storage node is received, the target metadata is locked according to the modification instruction, and the target metadata is modified in the shared memory of the shared storage node; a data log of the target metadata operation is constructed, and the data log is written to the disk; the modified target metadata is copied to the memory pool, mounted to the task queue of the background metadata disk flushing thread, and the modified target metadata is unlocked; the modified target metadata in the memory pool is flushed by the background metadata disk flushing thread according to the task order mounted in the task queue; after the modified target metadata disk flushing is completed, the data log in the disk is cleared. The embodiment of the present application reduces the lock holding time in the metadata modification process and reduces the lock conflict time. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0014] Figure 1This is a flowchart of a metadata asynchronous disk flushing method based on a database and computing separation architecture provided by an embodiment of the present application; Figure 2 It is a flowchart of a metadata asynchronous disk flushing method based on a database and computing separation architecture provided by another embodiment of the present application; Figure 3 It is a flowchart of a metadata asynchronous disk flushing method based on a database and computing separation architecture provided by another embodiment of the present application; Figure 4 It is a flowchart of a metadata asynchronous disk flushing method based on a database and computing separation architecture provided by another embodiment of the present application; Figure 5 It is a schematic diagram of the hardware structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0015] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0016] In the description of the present application, it should be understood that descriptions involving orientation, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0017] In the description of this application, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0018] In the description of this application, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in this application based on the specific content of the technical solution.

[0019] First, some nouns involved in this application are analyzed: Thread: It is the smallest unit that the operating system can schedule operations. Threads are contained in processes and are the actual operating units in processes. A thread refers to a single sequential control flow in a process. Multiple threads can run concurrently in a process, and each thread executes different tasks in parallel.

[0020] Database: is a computer software system that stores and manages data according to data structure. Database is an entity, it is a "warehouse" that can reasonably store data, users store transaction data to be managed in this "warehouse", the two concepts of "data" and "library" are combined to form a database. Database is an entity, it is a "warehouse" that can reasonably store data, users store transaction data to be managed in this "warehouse", the two concepts of "data" and "library" are combined to form a database.

[0021] Disk flushing: It is the process of writing data in memory to disk. It is an important mechanism in computer systems to ensure data persistence and prevent data loss. It uses different strategies and types to achieve reliable data storage.

[0022] The database computing separation architecture is an architectural design that deploys and manages data storage and data processing functions separately. Under this architecture, storage methods can be mainly divided into distributed storage and shared storage. The advantage of shared storage is that multiple computing nodes can access the same data together, reducing redundant data storage and reducing storage costs. However, in the prior art, since the master node needs to frequently append to files, it is necessary to modify the metadata first and then write the data, while the backup node frequently reads files and needs to read the metadata first and then read the data, resulting in frequent locking and unlocking operations and long lock waits, which affects the file operation speed and thus affects the performance of the overall storage and computing separation architecture.

[0023] The metadata asynchronous disk flushing method based on the database computing separation architecture provided in the embodiment of the present application is specifically illustrated through the following embodiments. First, the metadata asynchronous disk flushing method based on the database computing separation architecture in the embodiment of the present application is described.

[0024] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0025] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0026] The metadata asynchronous disk flushing method based on the database calculation separation architecture provided in the embodiment of the present application relates to the field of database management technology. The metadata asynchronous disk flushing method based on the database calculation separation architecture provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the metadata asynchronous disk flushing method based on the database calculation separation architecture, etc., but is not limited to the above forms.

[0027] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0028] It should be noted that in each specific implementation of the present application, when it is necessary to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0029] See also Figure 1 , Figure 1 A flowchart of a metadata asynchronous disk flushing method based on a database and computing separation architecture provided in an embodiment of the present application; Figure 1 As shown, an embodiment of the present application provides a metadata asynchronous disk flushing method based on a database computing separation architecture, including but not limited to steps S110-S150, and each step is introduced in turn below. Step S110: receiving a modification instruction of target metadata in a shared storage node, locking the target metadata according to the modification instruction, and performing a modification operation on the target metadata in a shared memory of the shared storage node; Step S120: construct a data log of the target metadata operation and write the data log to a disk; Step S130: copy the modified target metadata to the memory pool, mount it to the task queue of the background metadata flushing thread, and unlock the modified target metadata; Step S140: The background metadata flushing thread performs a flushing operation on the modified target metadata in the memory pool according to the order of tasks mounted in the task queue; Step S150: After the modified target metadata is flushed to disk, the data log in the disk is cleared.

[0030] In one embodiment, the separation architecture includes a shared storage node, a background metadata flushing thread and a memory pool. First, a modification instruction is received to indicate that the target metadata stored in the shared storage node needs to be modified, wherein the shared storage node refers to a storage area accessible to multiple processes or threads. Before modifying the target metadata, the target metadata needs to be locked to ensure the consistency and integrity of the data during the modification process. When the target metadata is locked, it can be modified in the shared storage node, which can reduce the copying of data between different processes and improve efficiency.

[0031] It should be noted that after modifying the target metadata, a data log is constructed to record the detailed information of this operation, such as the operation type, the data status before and after the modification, etc., so that the data can be restored in the event of a system crash or other failure. The modified target metadata is copied to the memory pool, which is a memory area used to temporarily store data to be written to the disk. At the same time, the operation of copying the target metadata to the memory pool will be added to the task queue of the background thread, waiting to be processed. The lock added before the target metadata is copied to the memory pool is released, allowing other processes or threads to access the target metadata. The background target metadata disk flushing thread is responsible for writing the data in the memory pool to the disk in the order of the task queue. This process is called "disk flushing", which is a process to ensure that data is transferred from volatile storage to non-volatile storage. When the target metadata is successfully written to the disk, the previously created data log is no longer needed and can be cleared to release storage space. The embodiment of the present application reduces the lock holding time in the metadata modification process and thus reduces the lock conflict time through an asynchronous metadata off-disk method.

[0032] It should be noted that locking the target metadata according to the modification instruction includes adding a disk lock and a shared memory lock, and unlocking the modified target metadata includes releasing the disk lock and the shared memory lock.

[0033] See also Figure 2 , Figure 2 A flowchart of another method for asynchronously flushing metadata based on a database and computing separation architecture provided in an embodiment of the present application; Figure 2 As shown, an embodiment of the present application provides a metadata asynchronous disk flushing method based on a database computing separation architecture, including but not limited to steps S210-S230, and each step is introduced in turn below. Step S210, dividing the disk into a plurality of regions according to a first granularity, and numbering the plurality of regions to obtain a plurality of region numbers; Step S220: divide the background metadata flushing thread into multiple threads, and number the multiple threads to obtain multiple thread numbers; Step S230: Obtain a target relationship table according to the multiple area numbers and the multiple thread numbers.

[0034] In one embodiment, the disk space is divided into multiple smaller areas, each of which can perform read and write operations independently. This division can be based on the physical characteristics or logical characteristics of the disk. The purpose of the division is to improve the efficiency of disk I / O operations and reduce I / O bottlenecks by processing data in different areas in parallel. Each area is assigned a unique number, called the area number, for easy management and identification.

[0035] In one embodiment, the background metadata flushing thread is responsible for writing data in the memory to the disk. To improve writing efficiency, these threads can be further subdivided into multiple sub-threads, each of which is responsible for processing a specific task or area. This multi-threaded processing can improve the system's concurrency capability, allowing multiple data write requests to be processed simultaneously. Each thread is assigned a unique number, called a thread number, for ease of management and scheduling.

[0036] In one embodiment, the table associates the region number of the disk with the thread number of the background thread. This relationship table can be used to schedule and allocate tasks to ensure that the data in each region can be processed by the correct thread. This mapping relationship can be static or dynamic.

[0037] In one embodiment, a 100-unit disk is divided into 5 partitions according to the first granularity of 20 units to obtain 5 area numbers, the background metadata flushing thread is divided into 3 threads, and the 3 threads are numbered to obtain 3 thread numbers, and the target relationship table is obtained according to the 5 area numbers and the 3 thread numbers. It should be noted that the system can recognize the partition where the target metadata is located.

[0038] In one embodiment, a 1000-unit disk is divided into 50 partitions according to a first granularity of 20 units to obtain 50 area numbers, the background metadata flushing thread is divided into 5 threads, and the 5 threads are numbered to obtain 5 thread numbers, and a target relationship table is obtained based on the 50 area numbers and the 5 thread numbers.

[0039] In one embodiment, when the target relationship table is obtained, an instruction to add target metadata is received, the thread number of the metadata is obtained according to the target table, and the target metadata is added to a preset thread according to the thread number.

[0040] It should be noted that the target relationship table is a mapping structure that defines the correspondence between the disk area number and the background metadata flushing thread number. When the system needs to add new target metadata, it will receive an add instruction. The system searches the target relationship table and finds the corresponding thread number according to the disk area number where the target metadata will be written, thereby determining that the target metadata will be processed by the corresponding thread. When the thread number corresponding to the target metadata is determined, the system will add the target metadata to the task queue of the thread. Each background thread has its own task queue for managing the target metadata that needs to be processed. The background thread will take the target metadata from its own task queue for processing, including but not limited to writing the target metadata to disk, updating the cache in memory, synchronizing the target metadata, and other operations. By assigning target metadata to different threads for processing, the efficiency of metadata addition operations can be significantly improved. Multiple threads can work in parallel and process target metadata in different areas at the same time, thereby reducing waiting time and improving overall performance.

[0041] See also Figure 3 , Figure 3 A flowchart of another method for asynchronously flushing metadata based on a database and computing separation architecture provided in an embodiment of the present application; Figure 3 As shown, an embodiment of the present application provides a metadata asynchronous disk flushing method based on a database computing separation architecture, including but not limited to steps S310-S330, and each step is introduced in turn below. Step S310, obtaining the number of regions and the number of threads according to the target relationship table; Step S320, calculating the number of regions and the number of threads to obtain a calculated value; Step S330: Obtain the thread number of the target metadata according to the calculated value.

[0042] In one embodiment, the system obtains the number of regions and the number of threads from the target relationship table, calculates the number of regions and the number of threads to obtain a calculated value, and then determines to allocate the target metadata to the corresponding thread according to the calculated value.

[0043] It should be noted that the calculation method for obtaining the calculated value includes but is not limited to weighted calculation, loop calculation, and mapping calculation. The embodiment of the present application does not impose any specific restrictions on the calculation method for obtaining the calculated value.

[0044] In one embodiment, when the number value of regions is 50 and the number value of threads is 3, the thread numbers of the threads are 0, 1, and 2. The number value of regions and the number value of threads are calculated, that is, the number value of regions 50 is used as the dividend and the number value of threads 3 is used as the divisor, and the remainder is 2, so the metadata located at region number 50 will be placed in the thread number 2.

[0045] In one embodiment, when the number value of regions is 100 and the number value of threads is 3, the thread numbers of the threads are 0, 1, and 2. The number value of regions and the number value of threads are calculated, that is, the number value of regions 100 is used as the dividend and the number value of threads 3 is used as the divisor. The remainder is 1, and the metadata located at region number 100 will be placed in the thread number 1.

[0046] In one embodiment, the number of regions is used as the dividend and the number of threads is used as the divisor for calculation to obtain the remainder, and the target metadata located at the region number will be placed in the thread whose thread number is the remainder.

[0047] See also Figure 4 , Figure 4 A flowchart of another method for asynchronously flushing metadata based on a database and computing separation architecture provided in an embodiment of the present application; Figure 4 As shown, an embodiment of the present application provides a metadata asynchronous disk flushing method based on a database computing separation architecture, including but not limited to steps S410-S430, and each step is introduced in turn below. Step S410: Create a chain of target metadata and put the target metadata into the chain to form a historical chain and obtain the first version of the target metadata; Step S420: Put the target metadata into the head of the history chain to form an intermediate chain and obtain the second version of the target metadata, wherein the second version of the target metadata points to the first version of the target metadata.

[0048] In one embodiment, the system creates a linked list for storing historical versions of the target metadata. This linked list is called a historical chain, which can track changes in the target metadata over time. When new target metadata is created, it is added to the historical chain and becomes the first node in the chain, that is, the first version of the target metadata. New target metadata changes are created and added to the head of the historical chain to form a second node and a chain. This chain is called an intermediate chain. The second node is the second version of the target metadata and points to the first version of the target metadata. By maintaining the historical chain of the target metadata, the system can track the change history of each target metadata item.

[0049] In one embodiment, when the background metadata flushing thread includes target metadata, the target metadata is placed at the head of the intermediate chain to form a target chain and a third version of the target metadata is obtained, and the third version of the target metadata points to the second version of the target metadata.

[0050] It should be noted that when the target metadata changes, a new target metadata version is created and added to the head of the intermediate chain, forming a third node and a chain. This chain is called the target chain. The third node is the third version of the target metadata and points to the second version of the target metadata. This operation is performed on the basis of the intermediate chain, which is a linked list formed in the previous version control process. The target chain is an extension of the history chain, which contains the latest target metadata version. Each time the target metadata is updated, a new version is added to the head of the chain, so that the latest target metadata can be quickly accessed while maintaining access to historical versions. The third version of the target metadata contains a pointer to the second version, thus forming a chain structure that allows the system to trace back to any previous version, ensuring that the history of target metadata changes can be tracked and restored.

[0051] In one embodiment, after obtaining the first version of the target metadata, the second version of the target metadata, and the third version of the target metadata, the first version of the target metadata or the first version of the target metadata and the second version of the target metadata on the disk are cleared, and the history chain or the intermediate chain is cleared.

[0052] In one embodiment, the history chain and the intermediate chain are data structures for storing target metadata versions. The history chain contains historical versions of the target metadata, while the intermediate chain contains the most recent versions. Deleting the nodes therein can free up storage space and update the pointer structure of the linked list to reflect the new linked list state.

[0053] In one embodiment, when the arrangement order of the chain of target metadata is the order of the serial numbers of the target metadata from large to small, each time the target metadata is modified, the value of the serial number of the target metadata is changed.

[0054] In one embodiment, the target metadata chain is arranged from large to small according to the serial number, indicating that the head of the chain (chain head) is the latest target metadata version, and the tail of the chain is the earliest version. This arrangement makes it very convenient to obtain the latest version of the target metadata. The serial number is a unique identifier of the target metadata version, which is used to distinguish different versions. Each time the target metadata is modified and a new version is generated, the serial number will be updated to reflect that this is a new version. Whenever the target metadata is modified, the system will create a new target metadata version and add it to the head of the chain. This new version will have a new serial number, which is larger than the serial numbers of all other versions in the chain to maintain the order of the chain.

[0055] The embodiment of the present application also provides a computer device, which includes: at least one memory, at least one processor, and at least one computer program, wherein the at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement any of the partition pruning methods of the database in the above embodiments. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0056] See also Figure 5 , Figure 5 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes: The processor 510 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 520 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 520 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 520, and the processor 510 calls and executes the metadata asynchronous disk flushing method based on the database and calculation separation architecture of the embodiment of this application; Input / output interface 530, used to implement information input and output; Communication interface 540, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); bus 550 , which transmits information between the various components of the device (e.g., processor 510 , memory 520 , input / output interface 530 , and communication interface 540 ); The processor 510 , the memory 520 , the input / output interface 530 and the communication interface 540 are connected to each other in communication within the device via the bus 550 .

[0057] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned metadata asynchronous disk flushing method based on the database and computing separation architecture is implemented.

[0058] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0059] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0060] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0061] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0062] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0063] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0064] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0065] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0066] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0067] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0068] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0069] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A metadata asynchronous disk flushing method based on a database and computing separation architecture, characterized in that: The separation architecture includes a shared storage node, a background metadata flushing thread and a memory pool, and the method includes: receiving a modification instruction of the target metadata in the shared storage node, locking the target metadata according to the modification instruction, and performing a modification operation on the target metadata in the shared memory of the shared storage node; Constructing a data log of the target metadata operation and writing the data log to a disk; The modified target metadata is copied to the memory pool, mounted to the task queue of the background metadata flushing thread, and the modified target metadata is unlocked; The modified target metadata in the memory pool is flushed to disk by the background metadata flushing thread according to the order of tasks mounted in the task queue; After the modified target metadata is flushed to disk, the data log in the disk is cleared.

2. The method according to claim 1, characterized in that The method further comprises: Dividing the disk into a plurality of regions according to a first granularity, and numbering the plurality of regions to obtain a plurality of region numbers; Divide the background metadata disk flushing thread into multiple threads, and number the multiple threads to obtain multiple thread numbers; A target relationship table is obtained according to the plurality of area numbers and the plurality of thread numbers.

3. The method according to claim 2, characterized in that In the case of obtaining the target relationship table, the method includes: An instruction for adding the target metadata is received, a thread number of the target metadata is obtained according to the target relationship table, and the target metadata is added to a preset thread according to the thread number.

4. The method according to claim 3, characterized in that The obtaining the thread number of the metadata according to the target relationship table includes: Obtaining the quantity value of the regions and the quantity value of the threads according to the target relationship table; Calculating the number value of the regions and the number value of the threads to obtain a calculated value; The thread number of the target metadata is obtained according to the calculated value.

5. The method according to claim 1, characterized in that The method further comprises: Creating a chain of the target metadata and placing the target metadata into the chain to form a history chain and obtaining a first version of the target metadata; The target metadata is placed at the head of the history chain to form an intermediate chain and a second version of the target metadata is obtained, wherein the second version of the target metadata points to the first version of the target metadata.

6. The method according to claim 5, characterized in that The method further comprises: When the background metadata flushing thread includes the target metadata, the target metadata is placed at the head of the intermediate chain to form a target chain and a third version of the target metadata is obtained, and the third version of the target metadata points to the second version of the target metadata.

7. The method according to claim 6, characterized in that When the first version of the target metadata, the second version of the target metadata, and the third version of the target metadata are obtained, the method further includes: After the data log is written to the disk, the first version of the target metadata or the first version of the metadata and the second version of the metadata in the disk are cleared, and the history chain or the intermediate chain is cleared.

8. The method according to claim 5, characterized in that In the case where the arrangement order of the chain of the target metadata is the order of the serial numbers of the target metadata from large to small, it includes: Each time the target metadata is modified, the value of the sequence number of the target metadata is changed.

9. A computer device, characterized in that: include: at least one memory; at least one processor; at least one computer program; The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement: the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to make a computer execute: the method according to any one of claims 1 to 8.

Images