Metadata storage method and apparatus

Abstract

The application provides a metadata storage method and device, the method comprising: receiving a storage request of metadata of a message queue; merging the metadata of the message queue into a sorted string table (SST) file; and calling a key-value (KV) engine to store the SST file in a target storage location of a storage device, wherein the KV engine adopts parallel append writing on the storage device. In this way, the read-write performance of the metadata of the message queue is improved by merging the metadata into the SST file and using the KV engine to append write the storage device.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and in particular to a method and apparatus for storing metadata. Background Technology

[0002] Message queues are an important component of distributed systems, typically used to address issues such as application coupling, asynchronous messaging, and traffic shaping. As more and more applications adopt message queue processing mechanisms, the storage of message queue metadata urgently needs to be addressed.

[0003] In related technologies, message queue metadata is typically stored on the server's local disk. When the message queue metadata stored on the local disk needs maintenance, operations such as data reading and writing, data deletion, and updating consumer points can be performed through the local file system.

[0004] However, if the message queue's metadata is stored on the server's local disk, each metadata change requires a full update and persistence to the local file system, followed by serialization of the message queue's metadata file and persistence to the local disk. When the message queue's metadata is large, there may be issues with low read / write performance. Summary of the Invention

[0005] This application provides a method and apparatus for storing metadata to solve the problem of low read and write performance of metadata in message queues in the prior art.

[0006] In a first aspect, embodiments of this application provide a method for storing metadata, the method comprising:

[0007] Receive storage requests for metadata from the message queue;

[0008] The metadata of the message queue is merged into a finite sorted string table SST file;

[0009] The key-value (KV) engine is invoked to store the SST file in the target storage location of the storage device. The KV engine writes to the storage device in a parallel append manner.

[0010] In an optional implementation, before merging the metadata of the message queue into a finitely sorted string table (SST) file, the method further includes:

[0011] Determine whether the amount of metadata in the message queue exceeds a data volume threshold;

[0012] The step of merging the metadata of the message queue into a finitely sorted string table (SST) file includes:

[0013] If the amount of metadata in the message queue is greater than the data volume threshold, then the metadata of the message queue will be merged into a finite sorted string table (SST) file.

[0014] In one optional implementation, merging the metadata of the message queue into a finitely sorted string table (SST) file includes:

[0015] The log-based structure of the KV engine is invoked to merge the LSM tree, and the metadata of the message queue is merged into a finite sorted string table SST file;

[0016] The LSM tree is used to convert the metadata of the message queue from random writing to sequential writing.

[0017] In an optional implementation, after determining whether the amount of metadata of the message queue is greater than a data volume threshold, the method further includes:

[0018] If the amount of metadata of the message queue is less than or equal to the data amount threshold, the message queue model in the distributed message middleware engine is invoked to store the SST file corresponding to the metadata of the message queue in the local file system.

[0019] In an optional implementation, before invoking the key-value (KV) engine to store the SST file at the target storage location on the storage device, the method further includes:

[0020] The target storage location is determined based on the message queue's title data, the message queue's identifier, and the message queue's offset data.

[0021] In an optional implementation, after invoking the key-value engine to store the SST file at the target storage location on the storage device, the method further includes:

[0022] Based on the storage time and range information of the SST file, add a deletion flag to the SST file;

[0023] The KV engine's filter is invoked to delete SST files containing the deletion identifier.

[0024] In one alternative implementation, the metadata of the message queue includes the index data of the message queue.

[0025] Secondly, embodiments of this application provide a metadata storage device, the device comprising:

[0026] The receiving module is used to receive storage requests for metadata from the message queue.

[0027] The processing module is used to merge the metadata of the message queue into a finite sorted string table (SST) file;

[0028] The storage module is used to call the key-value (KV) engine to store the SST file in the target storage location of the storage device. The KV engine writes to the storage device in a parallel append-only manner.

[0029] In one optional implementation, the processing module is specifically used to determine whether the amount of metadata of the message queue is greater than a data volume threshold; if the amount of metadata of the message queue is greater than the data volume threshold, then the metadata of the message queue is merged into a finite sorted string table (SST) file.

[0030] In one optional implementation, the processing module is specifically used to call the log-based structure of the KV engine to merge the LSM tree and merge the metadata of the message queue into a finite sorted string table (SST) file.

[0031] The LSM tree is used to convert the metadata of the message queue from random writing to sequential writing.

[0032] In one optional implementation, the processing module is further configured to, if the amount of metadata of the message queue is less than or equal to the data amount threshold, invoke the message queue model in the distributed message middleware engine to store the metadata of the message queue in the local file system.

[0033] In one optional implementation, the processing module is further configured to determine the target storage location based on the message queue's header data, the message queue's identifier, and the message queue's offset data.

[0034] In one optional implementation, the processing module is further configured to add a deletion identifier to the SST file based on the storage time information and range information of the SST file; and call the filter of the KV engine to delete the SST file containing the deletion identifier.

[0035] In one alternative implementation, the metadata of the message queue includes the index data of the message queue.

[0036] Thirdly, this application also provides an electronic device, including: a processor and a memory; the memory being used to store a computer program of the processor; the processor being configured to implement any of the possible methods in the first aspect by executing the computer program.

[0037] Fourthly, the present invention also provides a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing any of the possible methods of the first aspect.

[0038] Fifthly, the present invention also provides a computer program that causes a computer to perform the method as described in the first aspect.

[0039] In a sixth aspect, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in the first aspect above and in various possible designs of the first aspect.

[0040] This application provides a method and apparatus for storing metadata. First, a server receives a storage request for metadata from a message queue. Then, the server merges the metadata of the message queue into a finite sorted string table (SST) file. Finally, the server invokes a key-value (KV) engine to store the SST file in a target storage location on a storage device. The KV engine performs parallel append writing on the storage device. This method allows small files to be merged into an SST file, thus resolving the read / write bottleneck of a large number of small files. Furthermore, using a KV engine to append SST files to the target storage location in parallel reduces the allocation and serialization of large objects, lowering the memory reclamation pressure on large objects. Based on this, the metadata storage method and apparatus provided in this application improve the read / write performance of metadata in a message queue. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A schematic diagram illustrating an application scenario of a metadata storage method provided in this application embodiment;

[0043] Figure 2 A flowchart illustrating a method for storing metadata provided in an embodiment of this application;

[0044] Figure 3 A flowchart illustrating another method for storing metadata provided in an embodiment of this application;

[0045] Figure 4 A flowchart illustrating another method for storing metadata provided in an embodiment of this application;

[0046] Figure 5 A schematic diagram of the structure of a metadata storage device provided in an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0050] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0051] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0052] The technical names involved in the embodiments of this application will be explained below.

[0053] KV Engine: A component that provides high-performance reading and writing of KV data structures.

[0054] Production terminal: A terminal device that produces messages to a message queue.

[0055] Consumer terminal: A terminal device that retrieves messages from a consumer queue for consumption.

[0056] Message: A type of data information that carries communication.

[0057] LMS tree: A storage structure that converts random writes into sequential writes.

[0058] SST file: A file of arbitrary length, sorted by key value.

[0059] Message queues are an important component of distributed systems, typically used to address issues such as application coupling, asynchronous messaging, and traffic shaping. As more and more applications adopt message queue processing mechanisms, the storage of message queue metadata urgently needs to be addressed.

[0060] Below are two common methods for storing message queue metadata.

[0061] In the first message queue storage method, the message queue metadata is typically stored on the server's local disk. When the message queue metadata stored on the local disk needs maintenance, operations such as data reading and writing, data deletion, and updating consumer points can be performed through the local file system.

[0062] When the metadata of a message queue in a distributed message middleware is stored on the local disk via the local file system, multiple loading methods are supported, such as NIOFS, MMAP, SIMPLEFS, and SMB. In some cases, the message queue's metadata file can also be directly locked into an MMAP in memory.

[0063] Distributed message middleware can include, for example, RocketMQ and Kafka.

[0064] However, storing message queue metadata on the server's local disk may present the following problems:

[0065] First, each metadata change requires a full update and persistence to the local file system, followed by JSON serialization of the message queue's metadata file and persistence to the local disk. Because the large object allocations described above place significant pressure on the message queue system's memory garbage collection, the read / write performance of the message queue's metadata is low.

[0066] Secondly, due to the message queue model using distributed message middleware, when there is a lot of metadata in the message queue, single-machine use of MMAP mapping for file reading and writing has problems such as file number limitation and a sharp drop in read and write performance caused by a large number of small files.

[0067] Secondly, storing message queue metadata on the server's local disk results in a longer failure recovery time for the messaging system. Furthermore, a large number of message queues will manifest as numerous file descriptors in the local file system, thus consuming significant system resources.

[0068] In the second message queue storage method, the metadata of all message queues can be persistently stored in a centralized database node. However, making the database a single point of failure for the entire message queue system can easily create a single-point bottleneck. When dealing with a large number of small files, the performance of the database system becomes limited.

[0069] Therefore, both of the above message queue storage methods suffer from low read / write performance of message queue metadata.

[0070] To address the aforementioned issues, this application provides a method and apparatus for storing metadata. This involves merging the metadata of a message queue into a Sorted String Table (SST) file and then using a Key-Value (KV) engine to store the SST file at a target storage location on a storage device. This method resolves the bottleneck of reading and writing a large number of small files by merging small files into an SST file. Furthermore, using a KV engine to append the SST to the target storage location in parallel reduces the allocation and serialization of large objects, thus lowering the memory reclamation pressure on large objects. Based on this, the metadata storage method and apparatus provided in this application improve the read and write performance of message queue metadata.

[0071] The following describes the application scenarios of the metadata storage method involved in the embodiments of this application.

[0072] Figure 1 This is a schematic diagram illustrating an application scenario of a metadata storage method provided in an embodiment of this application. For example... Figure 1 As shown, terminal device 101 can access the metadata of the message queue in server 102. When terminal device 101 sends a metadata storage request to server 102, server 102 can merge the metadata of the message queue into an SST file and call the KV engine to store the SST file in the target storage location of server 102.

[0073] Among them, the terminal device 101 can be a tablet computer, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in a smart home, etc.

[0074] Server 102 can be, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers. Among them, cloud computing is a type of distributed computing, which is a super virtual computer composed of a group of loosely coupled computers.

[0075] It is understood that the above-mentioned method of storing metadata is implemented by the metadata storage device provided in the embodiments of this application. The metadata storage device may be part or all of a device, such as the above-mentioned server or the processor of the server.

[0076] The following uses a server with integrated or installed relevant executable code as an example to provide a detailed description of the technical solutions of the embodiments of this application. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0077] Figure 2 This is a flowchart illustrating a method for storing metadata provided in an embodiment of this application. This embodiment relates to the specific process by which a server stores the metadata of a message queue in a storage device. Figure 2 As shown, the methods for storing this metadata include:

[0078] S201, Receive the request to store the metadata of the message queue.

[0079] In this application, when a terminal device accesses the message queue of the access server, it can send a request to the server to store the metadata of the message queue.

[0080] It should be understood that the embodiments of this application do not limit the terminal device that sends the storage request for metadata of the message queue. In some embodiments, the terminal device can be a producer terminal or a consumer terminal. Specifically, a consumer terminal is a terminal device that can generate messages to the message queue, and a consumer terminal can be a terminal device that retrieves messages from the message queue for consumption.

[0081] Messages in the message queue are used to carry communication data.

[0082] Correspondingly, the metadata of the message queue is used to describe the messages in the message queue. It should be noted that the type of metadata to be stored is not limited in this embodiment of the application; for example, it may include index data, consumption location data, topic data, etc.

[0083] S202. Merge the metadata of the message queue into a finite sorted string table SST file.

[0084] In this step, when the server receives the metadata of the message queue sent by the terminal device, it can merge the metadata of the message queue into an SST file, thereby reducing the read and write pressure on the server's storage device.

[0085] The SST file is a file containing arbitrary lengths of data sorted by key values.

[0086] It should be understood that the embodiments of this application do not limit how the server merges the metadata of the message queue into a finitely sorted string table (SST) file. In some embodiments, the server may call the LogStructured Merge Tree (LSM) of the KV engine to merge the metadata of the message queue into a finitely sorted string table (SST) file.

[0087] Among them, the LSM tree is used to convert the metadata of the message queue from random writing to sequential writing, thereby greatly improving the writing speed and providing a high-efficiency KV engine.

[0088] It should be understood that the embodiments of this application do not limit the type of KV engine. In some embodiments, the KV engine may include levelDB, rocksDB, badger, etc.

[0089] In some embodiments, the server can also determine whether the amount of metadata (e.g., Topic data) of the message queue exceeds a data volume threshold. If the amount of metadata exceeds the threshold, the metadata is merged into a finite sorted string table (SST) file, and then the KV engine is invoked to append the SST file to the target storage location on the storage device. If the amount of metadata is determined to be less than or equal to the threshold, the message queue (Queue) model in the distributed message middleware engine is invoked to store the SST file corresponding to the metadata of the message queue in the local file system. Distributed message middleware can be, for example, RocketMQ or Kafka.

[0090] It should be noted that this application embodiment does not impose a limit on the data volume threshold, which can be set according to the actual situation. For example, when the metadata exceeds a preset quantity, a KV engine can be used to merge the metadata before storage. For example, when the metadata does not exceed the preset quantity, a Queue model in a distributed message middleware engine can be used for storage.

[0091] S203, invoke the key-value (KV) engine to store the SST file in the target storage location of the storage device.

[0092] In this step, after the server merges the message queue's metadata into a finite sorted string table (SST) file, it can call the KV engine to store the SST file in the target storage location on the storage device.

[0093] It should be understood that the embodiments of this application do not limit how the target storage location is determined. In some embodiments, the server can determine the target storage location based on the message queue title data, the message queue identifier (queue Id), and the message queue offset data.

[0094] The message queue's header data and identifier can be used to identify the message queue containing the metadata to be stored, while the offset data is used to determine the storage location of the metadata to be stored within the message queue's metadata.

[0095] It should be understood that the KV engine involved in the embodiments of this application performs parallel append writes on the storage device. By customizing the capabilities of KV engines such as RocksDB, it is made to only perform parallel append writes without updating the data in place, thereby greatly reducing the compaction cycle of the KV engine and improving the data reading capability.

[0096] Furthermore, since the KV engine appends the metadata of queued messages to the storage device in parallel, it is not necessary to strictly control the related operations of the distributed database, thereby improving the overall performance and throughput of writing metadata of queued messages in the messaging system.

[0097] It should be noted that the KV engine can use the file system to store the metadata of the message queue at its underlying level. When the metadata file is corrupted, the message queue on a single machine can be recovered through the commit log file.

[0098] In other embodiments, after the KV engine is invoked to store the SST file at the target storage location on the storage device, the server can also set the storage validity period and the range to be compacted for the SST file stored on the storage device. Based on the storage time information and range information of the SST file, a deletion flag is added to the SST file, thereby invoking the KV engine's compaction filter to delete the SST file containing the deletion flag.

[0099] The aforementioned filter is used to clean up expired data before the KV engine merges the metadata.

[0100] It should be understood that the present application does not limit the storage validity period of SST files, and can be set according to the actual situation. For example, it can be 1 day, 3 days, 7 days, etc.

[0101] For example, if the range of SST files to be normalized is 1-100, and the validity period is 7 days, then SST files stored for more than 7 days among the 1-100 SST files can be marked with a deletion flag, thereby calling the KV engine's filter to delete the SST files containing the deletion flag.

[0102] In this application, by setting storage expiration and range information for the metadata of queued messages, expired metadata can be deleted, thereby improving read and write performance. At the same time, it avoids the linear growth of small file data as topic data grows, ensuring system performance.

[0103] The metadata storage method provided in this application embodiment involves the server first receiving a metadata storage request from a message queue. Then, the server merges the metadata of the message queue into a finitely sorted string table (SST) file. Finally, the server invokes a key-value (KV) engine to store the SST file in the target storage location on a storage device. The KV engine performs parallel append writing on the storage device. This method allows small files to be merged into an SST file, thus resolving the read / write bottleneck of a large number of small files. Furthermore, using the KV engine to append the SST file to the target storage location in parallel reduces the allocation and serialization of large objects, lowering the memory reclamation pressure on large objects. Based on this, the metadata storage method and apparatus provided in this application embodiment improve the read / write performance of metadata for message queues.

[0104] Based on the above embodiments, the storage method of the metadata of the queue message can also be determined based on the amount of data. Figure 3 This is a flowchart illustrating another method for storing metadata provided in an embodiment of this application. Figure 3 As shown, the methods for storing this metadata include:

[0105] S301, Receive the storage request for the metadata of the message queue.

[0106] S302. Determine whether the amount of metadata in the message queue exceeds the data volume threshold.

[0107] If yes, proceed to step S304; otherwise, proceed to step S303.

[0108] S303. Call the message queue model in the distributed message middleware engine and store the SST file corresponding to the metadata of the message queue in the local file system.

[0109] S304. Merge the metadata of the message queue into a finite sorted string table SST file.

[0110] S305: Call the key-value (KV) engine to store the SST file in the target storage location on the storage device. The KV engine writes to the storage device in a parallel append manner.

[0111] The technical terms, technical effects, technical features, and optional implementation methods of steps S301 to S305 can be found in [reference]. Figure 2 The steps S201 to S203 shown are understood, and repeated information will not be described again here.

[0112] Based on the above embodiments, the metadata storage method provided in this application also includes periodically cleaning up the stored metadata. Figure 4 This is a flowchart illustrating another method for storing metadata provided in an embodiment of this application. Figure 4 As shown, the methods for storing this metadata include:

[0113] S401, Receive the storage request for the metadata of the message queue.

[0114] S402. Merge the metadata of the message queue into a finite sorted string table SST file.

[0115] S403: The key-value (KV) engine is invoked to store the SST file in the target storage location on the storage device. The KV engine writes to the storage device in a parallel append manner.

[0116] The technical terms, technical effects, technical features, and optional implementation methods of steps S401 to S403 can be found in [reference]. Figure 2 The steps S201 to S203 shown are understood, and repeated information will not be described again here.

[0117] S404. Based on the storage time and range information of the SST file, add a deletion marker to the SST file.

[0118] It should be understood that the present application does not limit the storage validity period of SST files, and can be set according to the actual situation. For example, it can be 1 day, 3 days, 7 days, etc.

[0119] In some embodiments, the aforementioned deletion identifier may specifically include a tombstone identifier. This identifier can be added when the message queue's metadata is merged into an SST and stored on a storage device. The server checks the tombstone identifiers of the SSTs stored on the storage device at preset time intervals (e.g., 10 minutes, 30 minutes, 1 hour, etc.) or at preset time points (e.g., 24:00 every day). If it is determined that the SST file has exceeded its storage validity period and is within the range to be regulated as indicated by the range information, the SST file can be deleted.

[0120] In other embodiments, the server may also set files of interest to the user outside the range indicated by the range information, thereby saving the files for a long period of time.

[0121] In this application, by setting a storage expiration time for the metadata of queued messages, expired metadata can be deleted, thereby improving read and write performance. At the same time, it avoids the linear growth of small file data as topic data grows, ensuring system performance.

[0122] S405: Call the KV engine's filter to delete SST files containing deletion flags.

[0123] The filters mentioned above are used to clean up expired data before the metadata is merged.

[0124] The metadata storage method provided in this application embodiment involves the server first receiving a metadata storage request from a message queue. Then, the server merges the metadata of the message queue into a finitely sorted string table (SST) file. Finally, the server invokes a key-value (KV) engine to store the SST file in the target storage location on a storage device. The KV engine performs parallel append writing on the storage device. This method allows small files to be merged into an SST file, thus resolving the read / write bottleneck of a large number of small files. Furthermore, using the KV engine to append the SST file to the target storage location in parallel reduces the allocation and serialization of large objects, lowering the memory reclamation pressure on large objects. Based on this, the metadata storage method and apparatus provided in this application embodiment improve the read / write performance of metadata for message queues.

[0125] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0126] Figure 5 This is a schematic diagram of a metadata storage device provided in an embodiment of this application. The metadata storage device can be implemented through software, hardware, or a combination of both to execute the metadata storage method described in the above embodiments. Figure 5 As shown, the metadata storage device 500 includes: a receiving module 501, a processing module 502, and a storage module 503.

[0127] In one optional implementation, the processing module is specifically used to determine whether the amount of metadata of the message queue is greater than a data volume threshold; if the amount of metadata of the message queue is greater than the data volume threshold, the metadata of the message queue is merged into a finite sorted string table SST file.

[0128] In one optional implementation, the processing module is specifically used to call the log-based structure of the KV engine to merge the LSM tree and merge the metadata of the message queue into a finite sorted string table SST file.

[0129] The LSM tree is used to convert the metadata of the message queue from random writing to sequential writing.

[0130] In one optional implementation, the processing module is further configured to, if it is determined that the amount of metadata of the message queue is less than or equal to a threshold, invoke the message queue model in the distributed message middleware engine to store the metadata of the message queue in the local file system.

[0131] In one optional implementation, the processing module is further configured to determine the target storage location based on the message queue header data, the message queue identifier, and the message queue offset data.

[0132] In one optional implementation, the processing module is further configured to add a deletion flag to the SST file based on the storage time information and range information of the SST file; and call the KV engine's filter to delete the SST file containing the deletion flag.

[0133] In one alternative implementation, the metadata of the message queue includes the message queue's index data.

[0134] It needs to be explained that, Figure 5 The metadata storage device provided in the embodiments can be used to execute the metadata storage method provided in any of the above embodiments. The specific implementation and technical effects are similar, and will not be described again here.

[0135] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device may include: multiple processors 601 and memory 602. Figure 6 This refers to an electronic device using a processor as an example.

[0136] Memory 602 is used to store programs. Specifically, the program may include program code, which includes computer operation instructions and is configured as follows:

[0137] Receive storage requests for metadata from the message queue;

[0138] Merge the message queue metadata into a finite sorted string table SST file;

[0139] The key-value (KV) engine is invoked to store the SST file in the target storage location on the storage device. The KV engine writes to the storage device in a parallel append manner.

[0140] The memory 602 may include high-speed RAM memory, and may also include non-volatile memory, such as multiple disk drives.

[0141] The processor 601 is used to execute computer execution instructions stored in the memory 602 to implement the above-mentioned metadata storage method;

[0142] The processor 601 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0143] Optionally, in specific implementations, if the communication interface, memory 602, and processor 601 are implemented independently, they can be interconnected via a bus to complete communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not imply that there is only one bus or one type of bus.

[0144] Optionally, in a specific implementation, if the communication interface, memory 602, and processor 601 are integrated on a single chip, then the communication interface, memory 602, and processor 601 can communicate through an internal interface.

[0145] The electronic device provided in this application embodiment can be the server described above.

[0146] This application also provides a chip, including a processor and an interface. The interface is used to input and output data or instructions processed by the processor. The processor is used to execute the metadata storage method provided in the above method embodiments. This chip can be applied to a metadata storage device.

[0147] This application embodiment also provides a computer-readable storage medium, which may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a disk, or an optical disk. Specifically, the computer-readable storage medium stores program information, which is used for the aforementioned metadata storage method.

[0148] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the metadata storage method described above.

[0149] This application also provides a computer program that enables a computer to execute the above-described method for storing metadata.

[0150] This application also provides a queue message storage system, including a server and a terminal device, wherein the server executes the above-described metadata storage method.

[0151] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for storing metadata, characterized in that, The method includes: Receive a storage request for the metadata of a message queue, wherein the metadata of the message queue includes the index data of the message queue; Determine whether the amount of metadata in the message queue exceeds a data volume threshold; If the amount of metadata in the message queue is greater than the data amount threshold, the metadata of the message queue is merged into a finite sorted string table (SST) file. The merging is achieved by calling the log structure of the key-value (KV) engine to merge the LSM tree. The LSM tree is used to convert the metadata of the message queue from random writing to sequential writing. The key-value (KV) engine is invoked to store the SST file in the target storage location of the storage device. The KV engine writes to the storage device in a parallel append-only manner. This parallel append-only write method is achieved by customizing the KV engine to support only append-only writes and not in-place updates. Set the storage validity period and range information to be regularized for the SST files stored on the storage device. According to the storage time information and range information of the SST files, add a deletion identifier to the SST files and call the filter of the KV engine to delete the SST files containing the deletion identifier. The filter is used to clean up expired data before the KV engine merges the metadata. If the amount of metadata of the message queue is less than or equal to the data amount threshold, the message queue model in the distributed message middleware engine is invoked to store the SST file corresponding to the metadata of the message queue in the local file system.

2. The method according to claim 1, characterized in that, The step of merging the metadata of the message queue into the SST file includes: The log-based structure of the KV engine is invoked to merge the LSM tree, and the metadata of the message queue is merged into the SST file.

3. The method according to claim 1, characterized in that, Before invoking the key-value (KV) engine to store the SST file at the target storage location on the storage device, the method further includes: The target storage location is determined based on the message queue's title data, the message queue's identifier, and the message queue's offset data.

4. A metadata storage device, characterized in that, The device includes: A receiving module is used to receive storage requests for the metadata of a message queue, wherein the metadata of the message queue includes the index data of the message queue; The processing module is used to determine whether the amount of metadata in the message queue is greater than a data volume threshold. If the amount of metadata in the message queue is greater than the data amount threshold, the metadata of the message queue is merged into a finite sorted string table (SST) file. The merging is achieved by calling the log structure of the key-value (KV) engine to merge the LSM tree. The LSM tree is used to convert the metadata of the message queue from random writing to sequential writing. The storage module is used to call the key-value (KV) engine to store the SST file in the target storage location of the storage device. The KV engine writes to the storage device in a parallel append-only manner. The parallel append-only manner is achieved by customizing the KV engine to support only append-only writes and not in-place updates. Set the storage validity period and range information to be regularized for the SST files stored on the storage device. According to the storage time information and range information of the SST files, add a deletion identifier to the SST files and call the filter of the KV engine to delete the SST files containing the deletion identifier. The filter is used to clean up expired data before the KV engine merges the metadata. The processing module is further configured to, if the amount of metadata of the message queue is less than or equal to the data amount threshold, invoke the message queue model in the distributed message middleware engine to store the SST file corresponding to the metadata of the message queue in the local file system.

5. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 3.

Images