A message queue implementation method and a message queue system

Abstract

This application relates to a message queue implementation method and a message queue system. The method includes: storing messages produced by a producer in a message storage table of a database, the message storage table including at least message identifiers, delivery status, and sharding parameters; determining a management service process based on identifier records in a distributed lock table, the management service process including a pull thread and a delivery thread, the pull thread being used to retrieve target messages with a pending delivery status from the message storage table, and distributing the target messages to corresponding in-memory message queues according to sharding parameters and preset consumer metadata, the delivery thread being used to start push tasks according to the ascending order of message identifiers, and simultaneously updating the delivery status of the corresponding target message in the message storage table to "delivering"; if a confirmation instruction for the target message is received from the message consumer, the target message is deleted from the message storage table.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a message queue implementation method and a message queue system. Background Technology

[0002] With the popularization of cloud computing technology, the expansion of the platform's distributed architecture has led to a continuous increase in the number of system modules, and the functions undertaken by each module have become increasingly refined. In order to ensure the continuous iteration capability of the system, each functional module is required to have a high degree of independence, reduce the direct dependence between modules, and ensure that the upgrade, iteration or maintenance of a single module will not affect the normal operation of other modules, thereby improving the overall stability and development efficiency of the system.

[0003] In distributed systems, the decoupling between modules inevitably necessitates the need for data exchange and information synchronization. Traditional solutions often rely on synchronous network interface calls for data interaction between modules. While message queue middleware is widely used as the core component for asynchronous communication in distributed systems, it is primarily employed in scenarios involving massive message volumes and high concurrency. In scenarios with smaller message volumes, it still requires fixed server resources, memory, and storage space, resulting in redundant resource consumption. Summary of the Invention

[0004] Therefore, it is necessary to provide a message queue system and message processing method to address the aforementioned technical problems.

[0005] Firstly, this application provides a method for implementing a message queue. The method includes:

[0006] The messages produced by the producer are stored in a message storage table in the database. The message storage table includes at least message identifier, delivery status, and fragmentation parameters.

[0007] The management service process is determined based on the identifier record in the distributed lock table. The management service process includes a pull thread and a delivery thread. The pull thread is used to retrieve the target message with the pending delivery status from the message storage table and distribute the target message to the corresponding memory message queue according to the sharding parameters and preset consumer metadata. The delivery thread is used to start the push task according to the message identifier in ascending order and update the delivery status of the corresponding target message in the message storage table to "delivering".

[0008] If an acknowledgment instruction for the target message is received from the message consumer, the target message is deleted from the message storage table.

[0009] In one embodiment, determining the management service process based on the identifier record in the distributed lock table includes:

[0010] Retrieve the lock identifier record from the distributed lock table;

[0011] The process that successfully inserts the lock identifier record is identified as the management service process, and the remaining processes are identified as backup service processes.

[0012] In one embodiment, after confirming the process that successfully inserted the lock identifier record as the management service process, the process includes:

[0013] The heartbeat records of the management service process in the heartbeat table are retrieved periodically. If the heartbeat table is not updated within a preset period and the heartbeat timestamp of the management service process is detected to be more than a preset time away from the current time, the backup service process is triggered to perform the lock identifier record operation on the distributed lock table.

[0014] In one embodiment, after determining the management service process based on the identifier record in the distributed lock table, the process includes:

[0015] The message topic of the target message is obtained. The message topic is used to retrieve the number of consumers and the number of memory message queues of each consumer from the preset consumer metadata. The hash value is obtained based on the sharding parameters of the target message. The modulo is taken with the number of consumers and the number of memory message queues of the corresponding consumers. Target messages with the same sharding parameters are distributed to the same memory message queue of the same consumer.

[0016] In one embodiment, the confirmation instruction is a success result code returned by the consumer;

[0017] If no success result code is received or the connection fails, the consumption is deemed to have failed. The delivery status of the target message is then updated to delivery failure, and the number of retries is incremented.

[0018] In one embodiment, the message storage table includes a consumer identifier field;

[0019] If the target message corresponds to multiple consumers, after each consumer successfully consumes the message, the consumer identifier is updated. When the field contains all consumer identifiers, it is determined that all consumers have successfully consumed the message, and the deletion operation is performed.

[0020] Secondly, this application also provides a message queue system applied to a database management platform, the system including a message queue service module, a message storage module and a coordination module;

[0021] The message storage module is a message storage table in the business relational database of the database management platform;

[0022] The message queue service module is deployed in the microservice cluster of the database management platform. The message queue service module includes a message receiving sub-service module and a message delivery sub-service module. The message receiving sub-service module is used to receive messages and write them to the message storage table. The message delivery sub-service module is used to cache the target messages with target status in the message storage table to local memory and allocate the target messages in local memory to the delivery process.

[0023] The coordination module is used to determine the management service process based on the identifier record in the distributed lock table. The management service process is used to execute message push tasks. The distributed lock table is created in a business relational database.

[0024] Thirdly, this application also provides an implementation apparatus for a message queue, the apparatus comprising:

[0025] The storage module is used to store the messages produced by the producer into a message storage table in the database. The message storage table includes at least message identifier, delivery status, and fragmentation parameters.

[0026] The execution module is used to determine the management service process based on the identifier record in the distributed lock table. The management service process includes a pull thread and a delivery thread. The pull thread is used to retrieve the target message with the pending delivery status from the message storage table and distribute the target message to the corresponding memory message queue according to the sharding parameters and preset consumer metadata. The delivery thread is used to start the push task according to the message identifier in ascending order and update the delivery status of the corresponding target message in the message storage table to "delivering".

[0027] The confirmation module is used to delete the target message from the message storage table if it receives a confirmation instruction for the target message returned by the message consumer.

[0028] Fourthly, this disclosure also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of a message queue implementation method.

[0029] Fifthly, this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of a message queue implementation method.

[0030] Sixthly, this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of a message queue implementation method.

[0031] The above-described message queue implementation method has at least the following beneficial effects:

[0032] The embodiment provided in this disclosure presents a message queue service that is not a standalone module but is directly embedded into the existing microservice development framework. Message communication between microservices can directly call the framework's built-in message queue interface, eliminating the need for additional third-party middleware integration and reducing development and maintenance costs. Reliable message delivery capabilities are achieved without significantly increasing the overall system cost. A relational database is used as the storage engine, allowing full reuse of the platform's business database without requiring additional computing and storage resources for the message queue.

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments or conventional technologies of this disclosure, the accompanying drawings used in the description of the embodiments or conventional technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is an application environment diagram of a message queue implementation method in one embodiment;

[0036] Figure 2 This is a flowchart illustrating the implementation method of a message queue in one embodiment;

[0037] Figure 3 This is a schematic diagram of a delivery thread group in one embodiment;

[0038] Figure 4 Here is a message data link diagram for one embodiment;

[0039] Figure 5 This is a structural block diagram of a message queue implementation device in one embodiment;

[0040] Figure 6 This is an internal structural diagram of a computer device in one embodiment;

[0041] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0042] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0043] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims. The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of other identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any specific order.

[0044] This disclosure provides a method for implementing a message queue, which can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0045] In some embodiments of this disclosure, such as Figure 2 As shown, a method for implementing a message queue is provided. In one specific embodiment, the method may include the following steps:

[0046] S202: Store the messages produced by the producer in the message storage table of the database. The message storage table includes at least message identifier, delivery status, and fragmentation parameters.

[0047] A dedicated message storage table will be created in the database as a dedicated repository for messages, where all messages sent by producers will be stored.

[0048] Message identifiers are generated incrementally based on the order in which messages are written to the table. Delivery status includes pending delivery, in progress, completed, and failed delivery; the default status when a message is first written is pending delivery. The fragmentation parameter is the message's routing label, determining which delivery thread should handle the message, ensuring that messages of the same type are delivered in order.

[0049] Producers push the generated message content to the message queue, either by calling the message queue API via RPC or by directly writing the message to a storage table.

[0050] The message queue provides a standardized RPC interface. The interface parameters include core information such as message content and sharding parameters. When the producer calls this interface, it passes in the necessary parameters. After receiving the request, the message queue's receiving thread automatically executes the SQL INSERT statement to write the message to the message storage table in the database. After successful writing, it returns a push success message to the producer.

[0051] Producers have write permissions to the message storage table and can directly execute INSERT statements, including sharding parameters. They can leverage the transaction features of relational databases to place business data changes and message writing to the table within the same transaction, while the message identifier is still automatically generated by the database.

[0052] S204: Determine the management service process based on the identifier record in the distributed lock table. The management service process includes a pull thread and a delivery thread. The pull thread is used to retrieve the target message in the delivery status from the message storage table and distribute the target message to the corresponding memory message queue according to the sharding parameters and preset consumer metadata. The delivery thread is used to start the push task according to the message identifier in ascending order and update the delivery status of the corresponding target message in the message storage table to "delivering".

[0053] The message queue deploys multiple processes that compete for the lock. The process that successfully acquires the lock becomes the management service process, while the process that fails becomes a backup process. A dedicated distributed lock table exists in the database, containing a unique lock identifier. All processes simultaneously insert their lock identifier and their own process ID into this table. Because the lock identifier is unique, only one process can successfully insert the lock. The process that successfully inserts the lock becomes the management service process, responsible for all subsequent message delivery. This ensures that only one process manages message delivery at a time, preventing duplicate pushes and message order disruptions caused by multiple processes simultaneously pulling and delivering messages.

[0054] The fetch thread retrieves target messages with a delivery status of "awaiting delivery" from the message storage table in the database, and then distributes them to the corresponding in-memory message queues according to rules, providing temporary memory space for message storage. The message sharding parameters ensure that messages with the same sharding parameters are assigned to the same in-memory queue, maintaining the order of messages from the same user. Preset consumer metadata ensures that messages are evenly distributed across different in-memory queues, avoiding excessive pressure on any single queue. For example, a certain type of message might have 3 consumers, with each consumer having 2 in-memory queues.

[0055] The delivery thread retrieves messages sequentially from the corresponding in-memory message queue, pushes them to the consumer, and updates the message status simultaneously. Push tasks are initiated in ascending order of message identifiers, and messages are pushed to consumers via protocols such as HTTP. Simultaneously, SQL is executed to change the delivery status of the message in the database message storage table to "delivering".

[0056] Figure 3 This is a schematic diagram of a delivery thread group in one embodiment. It represents an ideal scenario where only one process runs the delivery thread. The message retrieval thread needs to retrieve all newly added messages from the message storage table, using general SQL queries to implement a relatively efficient message data retrieval strategy. Configuration is based on consumer metadata. Messages are distributed to different in-memory message queues according to the number of consumers and queuing queues configured by the message topic. Each in-memory consumption queue corresponds to a dedicated delivery thread. Each delivery thread is responsible for the sequential delivery of messages in one in-memory message queue. Consumers need to implement an HTTP protocol consumption interface as required, and the delivery thread confirms the message status based on the result code returned by the consumption interface.

[0057] S206: If an acknowledgment instruction for the target message is received from the message consumer, the target message is deleted from the message storage table.

[0058] Figure 4 This is a message data link diagram in one embodiment. After the message queue's delivery thread pushes the target message to the consumer, it waits for the consumer's response. Only after receiving a confirmation instruction from the consumer will the deletion operation be performed. After the confirmation instruction is verified, the message queue executes a SQL DELETE statement to completely delete the message from the message storage table in the database, releasing storage resources. If no confirmation instruction is received, the message will not be deleted and will be retried according to preset rules until a confirmation instruction is received or the retry limit is reached.

[0059] In the above message queue implementation method, the message queue service in this solution is not an independent module, but is directly embedded into the existing microservice development framework. Message communication between microservices can directly call the framework's built-in message queue interface, eliminating the need for additional third-party middleware integration and reducing development and maintenance costs. Reliable message delivery capabilities are achieved without significantly increasing the overall system cost. A relational database is used as the storage engine, allowing full reuse of the platform's business database without requiring additional computing and storage resources for the message queue.

[0060] In some embodiments of this disclosure, determining the management service process based on the identifier record in the distributed lock table includes:

[0061] Retrieve the lock identifier record from the distributed lock table;

[0062] The process that successfully inserts the lock identifier record is identified as the management service process, and the remaining processes are identified as backup service processes.

[0063] A dedicated distributed lock table is created in the database. This table contains at least one unique field: a lock identifier. Multiple processes are deployed in the message queue. Upon startup, each process simultaneously executes an SQL insert operation into the distributed lock table. Each process attempts to bind its process ID to the fixed lock identifier and write it to the lock table. Because the lock identifier is a unique field, the database rejects duplicate inserts. Only the first process to successfully insert the record can insert it; other processes will fail due to unique constraint conflicts. The process that successfully inserts the lock identifier becomes the sole management service process, responsible for starting the message fetching and delivery threads, and performing core tasks such as message fetching, distribution, and delivery. Backup service processes are all processes that failed to insert the lock identifier. They do not start delivery-related threads; they only run stateless message receiving threads and continuously monitor the lock identifier record and the management service process's heartbeat, waiting to take over the lock if the original manager fails.

[0064] In some embodiments of this disclosure, after confirming the process that successfully inserts the lock identifier record as a management service process, the process includes:

[0065] The heartbeat records of the management service process in the heartbeat table are retrieved periodically. If the heartbeat table is not updated within a preset period and the heartbeat timestamp of the management service process is detected to be more than a preset time away from the current time, the backup service process is triggered to perform the lock identifier record operation on the distributed lock table.

[0066] A dedicated heartbeat table is created in the database, with fields including lock identifier, management service process ID, and heartbeat timestamp. Once any process is identified as a management service process, a heartbeat update mechanism is initiated, automatically executing SQL at preset intervals to update the last heartbeat timestamp in the heartbeat table. Backup service processes periodically query the heartbeat table to obtain the last heartbeat timestamp of the current management service process and determine whether the management service process is functioning correctly.

[0067] If the heartbeat table is not updated within a preset period, and the last heartbeat timestamp is more than a preset time from the current time, the management service process is deemed to have failed. All backup processes will then attempt to insert a lock identifier record into the distributed lock table again. The process that successfully inserts the lock identifier record will be promoted to the new management service process. The new management service process will immediately start its pull and delivery threads, taking over the delivery tasks from the original management service process, and the heartbeat table will be updated with the information of the new management service process.

[0068] In some embodiments of this disclosure, after determining the management service process based on the identifier record in the distributed lock table, the process includes:

[0069] The message topic of the target message is obtained. The message topic is used to retrieve the number of consumers and the number of memory message queues of each consumer from the preset consumer metadata. The hash value is obtained based on the sharding parameters of the target message. The modulo is taken with the number of consumers and the number of memory message queues of the corresponding consumers. Target messages with the same sharding parameters are distributed to the same memory message queue of the same consumer.

[0070] A message topic can be understood as a category tag for a message. When a producer sends a message, it adds this tag and writes it to the message storage table. After the pull thread retrieves the target message from the message storage table, it first reads its message topic. Using this topic, it searches the pre-defined consumer metadata to find the specific consumption configuration rules for that topic. For example, an order creation message might correspond to 3 consumers, each with 2 in-memory message queues. A hash operation is performed on the sharding parameters of the target message to obtain a fixed integer. This hash value is divided by the number of consumers found in the first step, and the remainder is the consumer index to which the message should be assigned. Then, the same hash value is divided by the number of in-memory message queues for each consumer, and the remainder is the queue index for that consumer. The pull thread then distributes the current target message to the in-memory message queue corresponding to the queue index of the consumer corresponding to the consumer index. As long as the message sharding parameters are the same, the calculated hash value and the remainder of the two modulo operations are exactly the same, ensuring that the message enters the same in-memory message queue of the same consumer.

[0071] In some embodiments of this disclosure, the confirmation instruction is a success result code returned by the consumer;

[0072] If no success result code is received or the connection fails, the consumption is deemed to have failed. The delivery status of the target message is then updated to delivery failure, and the number of retries is incremented.

[0073] A success result code is a standardized success response code returned by the consumer after receiving and processing a message. Once the delivery thread pushes a message to the consumer and receives this success result code, it considers the consumer to have successfully received and processed the message. If consumption fails, the delivery thread executes an SQL statement to change the delivery status of the target message in the message storage table from "delivered" to "delivered failed," preventing other threads from repeatedly pushing the message. The message storage table contains a retry count field to record how many times the message has failed; subsequent retries are performed based on this retry count with penalty time.

[0074] In some embodiments of this disclosure, the message storage table includes a consumer identifier field;

[0075] If the target message corresponds to multiple consumers, after each consumer successfully consumes the message, the consumer identifier is updated. When the field contains all consumer identifiers, it is determined that all consumers have successfully consumed the message, and the deletion operation is performed.

[0076] The consumer identifier field records which consumers have successfully processed the message. The delivery thread pushes the message to each consumer that has subscribed to it. When a consumer successfully processes the message and returns a success result code, the delivery thread executes an SQL update operation to add the consumer's identifier to the consumer identifier field. After each update to the consumer identifier field, the identifier in the field is compared with the list of all subscribed consumer identifiers. If the consumer identifier field contains the identifiers of all subscribed consumers, it is determined that all consumers have successfully processed the message.

[0077] In some embodiments of this disclosure, a message queue system is applied to a database management platform, the system including a message queue service module, a message storage module, and a coordination module;

[0078] The message storage module is a message storage table in the business relational database of the database management platform;

[0079] The message queue service module is deployed in the microservice cluster of the database management platform. The message queue service module includes a message receiving sub-service module and a message delivery sub-service module. The message receiving sub-service module is used to receive messages and write them to the message storage table. The message delivery sub-service module is used to cache the target messages with target status in the message storage table to local memory and allocate the target messages in local memory to the delivery process.

[0080] The coordination module is used to determine the management service process based on the identifier record in the distributed lock table. The management service process is used to execute message push tasks. The distributed lock table is created in a business relational database.

[0081] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0082] Based on the same inventive concept, this disclosure also provides an apparatus for implementing a message queue to implement the aforementioned method for implementing a message queue. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in the embodiments of the message queue implementation apparatus provided below can be found in the limitations of the message queue implementation method described above, and will not be repeated here.

[0083] The apparatus may include a system (including a distributed system), software (application), module, component, server, client, etc., that uses the methods described in the embodiments of this specification, combined with necessary hardware implementation. Based on the same innovative concept, the apparatuses in one or more embodiments provided in this disclosure are as described in the following embodiments. Since the implementation schemes and methods for solving the problem by the apparatus are similar, the implementation of the specific apparatus in the embodiments of this specification can refer to the implementation of the foregoing methods, and repeated details will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatuses described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0084] In one embodiment, such as Figure 5 As shown, a message queue implementation device 500 is provided. The device can be the aforementioned server, or a module, component, device, or unit integrated into the server. The device 500 may include:

[0085] Storage module 502 is used to store messages produced by the producer into a message storage table in the database. The message storage table includes at least message identifier, delivery status, and fragmentation parameters.

[0086] The execution module 504 is used to determine the management service process based on the identifier record in the distributed lock table. The management service process includes a pull thread and a delivery thread. The pull thread is used to retrieve the target message with the pending delivery status from the message storage table and distribute the target message to the corresponding memory message queue according to the sharding parameters and preset consumer metadata. The delivery thread is used to start the push task according to the message identifier in ascending order and update the delivery status of the corresponding target message in the message storage table to "delivering".

[0087] The confirmation module 506 is used to delete the target message from the message storage table if it receives a confirmation instruction for the target message returned by the message consumer.

[0088] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0089] The modules in the aforementioned message queue implementation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0090] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores messages. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a message queue implementation method.

[0091] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a message queue method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0092] Those skilled in the art will understand that Figure 6 , Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the computer device to which the present disclosure is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0093] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods described in any embodiment of this disclosure.

[0094] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in any embodiment of this disclosure.

[0095] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this disclosure may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this disclosure may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0096] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the appended claims.

Claims

1. A method of implementing a message queue, characterized by, The method includes: The messages produced by the producer are stored in the message storage table of the business relational database of the database management platform. The message storage table includes at least message identifier, delivery status and fragmentation parameters. The management service process is determined based on the identifier record in the distributed lock table. The management service process includes a pull thread and a delivery thread. The pull thread is used to retrieve the target message with the pending delivery status from the message storage table and distribute the target message to the corresponding memory message queue according to the sharding parameters and preset consumer metadata. The delivery thread is used to start the push task according to the message identifier in ascending order, and at the same time update the delivery status of the corresponding target message in the message storage table to "delivering". The distributed lock table is created in a business relational database. The step of determining the management service process based on the identifier record in the distributed lock table includes: Retrieve the lock identifier record from the distributed lock table; The process that successfully inserts the lock identifier record is identified as the management service process, and the remaining processes are identified as backup service processes. If a confirmation instruction for the target message is received from the consumer, the target message is deleted from the message storage table.

2. The method according to claim 1, characterized in that, After confirming the process that successfully inserts the lock identifier record as the management service process, the process includes: The heartbeat records of the management service process in the heartbeat table are retrieved periodically. If the heartbeat table is not updated within a preset period and the heartbeat timestamp of the management service process is detected to be more than a preset time away from the current time, the backup service process is triggered to perform the lock identifier record operation on the distributed lock table.

3. The method according to claim 1, characterized in that, After determining the management service process based on the identifier record in the distributed lock table, the process includes: The message topic of the target message is obtained. The message topic is used to retrieve the number of consumers and the number of memory message queues of each consumer from the preset consumer metadata. The hash value is obtained based on the sharding parameters of the target message. The modulo is taken with the number of consumers and the number of memory message queues of the corresponding consumers. Target messages with the same sharding parameters are distributed to the same memory message queue of the same consumer.

4. The method of claim 1, wherein, The confirmation instruction is a success result code returned by the consumer; If no success result code is received or the connection fails, the consumption is deemed to have failed. The delivery status of the target message is then updated to delivery failure, and the number of retries is incremented.

5. The method of claim 1, wherein, The message storage table includes a consumer identifier field; If the target message corresponds to multiple consumers, after each consumer successfully consumes the message, the consumer identifier is updated. When the field contains all consumer identifiers, it is determined that all consumers have successfully consumed the message, and the deletion operation is performed.

6. A message queue system, characterized by The system, applied to a database management platform, includes a message queue service module, a message storage module, and a coordination module. The message storage module is a message storage table in the business relational database of the database management platform; The message queue service module is deployed in the microservice cluster of the database management platform. The message queue service module includes a message receiving sub-service module and a message delivery sub-service module. The message receiving sub-service module is used to receive messages and write them to the message storage table. The message delivery sub-service module is used to cache target messages with target status in the message storage table to local memory and allocate the target messages in local memory to the delivery process. Message communication between microservices calls the built-in message queue interface of the framework, without the need for additional integration of third-party middleware. The coordination module is used to determine the management service process based on the identifier record in the distributed lock table. The management service process is used to execute message push tasks. The distributed lock table is created in a business relational database. Determining the management service process based on the identifier record in the distributed lock table includes: obtaining the operation of executing the lock identifier record in the distributed lock table; confirming the process that successfully inserts the lock identifier record as the management service process, and confirming the remaining processes as backup service processes.

7. An apparatus for implementing a message queue, the apparatus comprising: The device includes: The storage module is used to store the messages produced by the producer into the message storage table of the business relational database of the database management platform. The message storage table includes at least message identifier, delivery status, and fragmentation parameters. The execution module is used to determine the management service process based on the identifier record in the distributed lock table. The management service process includes a pull thread and a delivery thread. The pull thread is used to retrieve the target message with the pending delivery status from the message storage table and distribute the target message to the corresponding memory message queue according to the sharding parameters and preset consumer metadata. The delivery thread is used to start the push task according to the message identifier in ascending order, and at the same time update the delivery status of the corresponding target message in the message storage table to "delivering". The distributed lock table is created in a business relational database. The step of determining the management service process based on the identifier record in the distributed lock table includes: Retrieve the lock identifier record from the distributed lock table; The process that successfully inserts the lock identifier record is identified as the management service process, and the remaining processes are identified as backup service processes. The confirmation module is used to delete the target message from the message storage table if it receives a confirmation instruction for the target message returned by the consumer.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images