Message queue processing method and device, electronic equipment and storage medium

Abstract

The invention provides a message queue processing method and device, electronic equipment and a storage medium, and the method comprises the steps: creating at least one message table in a database, each message table corresponding to a target queue, and the message table being used for storing description information of messages; the message table comprises at least one data chain table, the data chain table is used for storing message content, and each data chain table corresponds to one sub-queue of the target queue; obtaining at least one target message produced by a producer and description information carried by each target message, wherein the description information of the target message comprises a user identifier of the target message; and distributing the target messages with the same user identifier to the same message table according to the user identifier of each target message. In this way, the real-time performance of service processing of other users cannot be affected through the two-dimensional queue mode. Besides, compared with a middleware mode, the mode of creating the message table in the database can reduce less hardware and maintenance cost.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a message queue processing method, a message queue processing device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the growth of business volume, more and more businesses, including some with real-time requirements, have been connected to the message queue (MQ) processing mechanism. Its specific implementation is: the producer program produces messages and puts them into the specified queue, and then the consumer program pulls the messages from the specified queue for consumption. At the same time, the consumption process is recorded through logs, and related messages are tracked based on the logs later.

[0003] However, the current message queue processing method is prone to cause message congestion when a user's concurrent message push volume is large, resulting in untimely message consumption by other users, which in turn affects the real-time processing of other users' business. In addition, the current message queue needs to be configured with middleware, which has high requirements for software and hardware environment, procurement cost, compatibility, etc. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a message queue processing method, a message queue processing device, an electronic device, and a computer-readable storage medium.

[0005] The present disclosure provides a message queue processing method, including: At least one message table is created in a database, each message table corresponds to a target queue, and the message table is used to store description information of the message; the message table includes at least one data linked table, and the data linked table is used to store message content, and each data linked table corresponds to a subqueue of the target queue; Acquire at least one target message produced by a producer and description information carried by each of the target messages, wherein the description information of the target message includes a user identifier of the target message; According to the user identification of each target message, the target messages with the same user identification are allocated to the same message table.

[0006] In an optional implementation manner, the description information of the target message further includes a data identifier, and according to the user identifier of each of the target messages, the target messages with the same user identifier are assigned to the same message table, including: For each target message, determining, according to the data identifier of the target message, whether the target message already exists in the same message table; If it exists, then add the target message to the data linked list corresponding to the existing target message; If not, the target message is assigned to a new data linked list in the same message table.

[0007] In an optional implementation manner, allocating target messages with the same user identifier to the same message table according to the user identifier of each target message includes: According to the user identifier of each target message, the target messages with the same user identifier are allocated to the same message table using a consistent hashing algorithm.

[0008] In an optional implementation, the description information of the target message includes a message type, and the message type includes a first type, a second type, and a third type. Different types indicate different consumption speeds, wherein the consumption speed of the first type is greater than the consumption speed of the second type, and the consumption speed of the second type is greater than the consumption speed of the third type; the method further includes: The at least one message table is searched from the database, and multiple consumer threads are started to consume the target messages in the at least one message table, the multiple consumer threads include a first type of consumer thread, a second type of consumer thread and a third type of consumer thread, the first type of consumer thread is used to consume the first type of target message, the second type of consumer thread is used to consume the second type of target message, and the third type of consumer thread is used to consume the third type of target message.

[0009] In an optional implementation, the first type refers to a type of consumption according to priority, and the second type and the third type refer to types of consumption according to dispatch time.

[0010] In an optional implementation, the number of the first-category consumer threads is greater than the number of the second-category consumer threads, and the number of the second-category consumer threads is greater than the number of the third-category consumer threads.

[0011] In an optional implementation manner, creating at least one message table in the database includes: At least one message table is created in the database according to the business data volume of the producer and the number of consumer threads.

[0012] In an optional implementation, when the target message is assigned to the message table, the type of the target message is defaulted to the first type, and the method further includes: For any target message, during the process of consuming the target message, the first repeated consumption times of the target message under the first type are counted. When the first repeated consumption times reaches a first preset threshold, the message type of the target message is changed from the first type to the second type.

[0013] In an optional implementation manner, the method further includes: The second repeated consumption times of the target message under the second type are counted, and when the second repeated consumption times reaches a second preset threshold, the message type of the target message is changed from the second type to the third type.

[0014] In an optional implementation, when the target message is assigned to the message table, the type of the target message is defaulted to the first type, and the method further includes: For any target message, if the dispatch waiting time of the target message under the first type is greater than a first time threshold, the message type of the target message is changed from the first type to the second type.

[0015] In an optional implementation manner, different users in the same message table have corresponding maximum consumption speeds respectively, and the method further includes: When the consumption speed of any target user in the same message table reaches the corresponding maximum consumption speed, the dispatching time of the target message of the target user is extended, and the number of retries of the target message of the target user is increased by 1.

[0016] In an optional implementation manner, starting multiple consumer threads to consume the target message in the at least one message table includes: A plurality of consumption threads are started to consume target messages in the at least one message table at a consumption speed increasing in sequence according to a preset time interval.

[0017] In an optional implementation manner, the description information of the target message further includes a message category; and the method further includes: During the process of consuming the target message in the at least one message table, if a failure of the target consumption service is detected, the consumption speed of the first target message of the message category corresponding to the target consumption service is adjusted from the current consumption speed to a preset minimum consumption speed.

[0018] In an optional implementation manner, the method further includes: When it is detected that the target consumption service returns to normal, the consumption speed of the first target message is adjusted in a step-by-step manner to a maximum consumption speed corresponding to the first target message.

[0019] In an optional implementation manner, the method further includes: In the process of consuming the target message in the at least one message table, if it is detected that the target consumption service enters a current limiting state, the consumption speed of the target message corresponding to the target consumption service is reduced in a step-by-step manner.

[0020] In an optional implementation, the step-by-step time intervals increase sequentially.

[0021] The present disclosure also provides a message queue processing device, including: A creation module, used to create at least one message table in the database, each message table corresponds to a target queue, and the message table is used to store the description information of the message; the message table includes at least one data linked table, and the data linked table is used to store the message content, and each data linked table corresponds to a subqueue of the target queue; A production module, used to obtain at least one target message produced by a producer and description information carried by each of the target messages, wherein the description information of the target message includes a user identifier of the target message; The configuration module is used to allocate the target messages with the same user identification to the same message table according to the user identification of each target message.

[0022] In an optional implementation manner, the description information of the target message further includes a data identifier, and the configuration module is specifically configured to: For each target message, determining, according to the data identifier of the target message, whether the target message already exists in the same message table; If it exists, then add the target message to the data linked list corresponding to the existing target message; If not, the target message is assigned to a new data linked list in the same message table.

[0023] In an optional implementation manner, the configuration module is specifically used to: According to the user identifier of each target message, the target messages with the same user identifier are allocated to the same message table using a consistent hashing algorithm.

[0024] In an optional implementation, the description information of the target message includes a message type, the message type includes a first type, a second type, and a third type, and different types indicate different consumption speeds, wherein the consumption speed of the first type is greater than the consumption speed of the second type, and the consumption speed of the second type is greater than the consumption speed of the third type; the device further includes: A consumption module is used to search the at least one message table from the database and start multiple consumption threads to consume the target messages in the at least one message table. The multiple consumption threads include a first type of consumption thread, a second type of consumption thread and a third type of consumption thread. The first type of consumption thread is used to consume the first type of target message, the second type of consumption thread is used to consume the second type of target message, and the third type of consumption thread is used to consume the third type of target message.

[0025] In an optional implementation, the first type refers to a type of consumption according to priority, and the second type and the third type refer to types of consumption according to dispatch time.

[0026] In an optional implementation, the number of the first-category consumer threads is greater than the number of the second-category consumer threads, and the number of the second-category consumer threads is greater than the number of the third-category consumer threads.

[0027] In an optional implementation manner, the creation module is specifically used to: At least one message table is created in the database according to the business data volume of the producer and the number of consumer threads.

[0028] In an optional implementation, when the target message is assigned to the message table, the type of the target message is set to the first type by default, and the configuration module is further configured to: For any target message, during the process of consuming the target message, the first repeated consumption times of the target message under the first type are counted. When the first repeated consumption times reaches a first preset threshold, the message type of the target message is changed from the first type to the second type.

[0029] In an optional implementation manner, the configuration module is further used to: The second repeated consumption times of the target message under the second type are counted, and when the second repeated consumption times reaches a second preset threshold, the message type of the target message is changed from the second type to the third type.

[0030] In an optional implementation, when the target message is assigned to the message table, the type of the target message is set to the first type by default, and the configuration module is further configured to: For any target message, if the dispatch waiting time of the target message under the first type is greater than a first time threshold, the message type of the target message is changed from the first type to the second type.

[0031] In an optional implementation manner, different users in the same message table have corresponding maximum consumption speeds respectively, and the configuration module is further used to: When the consumption speed of any target user in the same message table reaches the corresponding maximum consumption speed, the dispatching time of the target message of the target user is extended, and the number of retries of the target message of the target user is increased by 1.

[0032] In an optional implementation manner, the consumption module is specifically used for: A plurality of consumption threads are started to consume target messages in the at least one message table at a consumption speed increasing in sequence according to a preset time interval.

[0033] In an optional implementation manner, the description information of the target message further includes a message category; and the consumption module is further configured to: During the process of consuming the target message in the at least one message table, if a failure of the target consumption service is detected, the consumption speed of the first target message of the message category corresponding to the target consumption service is adjusted from the current consumption speed to a preset minimum consumption speed.

[0034] In an optional implementation manner, the consumption module is further used for: When it is detected that the target consumption service returns to normal, the consumption speed of the first target message is adjusted in a step-by-step manner to a maximum consumption speed corresponding to the first target message.

[0035] In an optional implementation manner, the consumption module is further used for: In the process of consuming the target message in the at least one message table, if it is detected that the target consumption service enters a current limiting state, the consumption speed of the target message corresponding to the target consumption service is reduced in a step-by-step manner.

[0036] In an optional implementation, the step-by-step time intervals increase sequentially.

[0037] An embodiment of the present disclosure also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of any possible message queue processing method described above are performed.

[0038] The embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any possible message queue processing method described above are executed.

[0039] The message queue processing method, device, electronic device and storage medium provided by the embodiments of the present disclosure create at least one message table in a database, the message table is used to store the description information of the message, each message table corresponds to a target queue, and the message table includes at least one data linked table, the data linked table is used to store the message content, and each data linked table corresponds to a sub-queue of the target queue. In this way, through a two-dimensional queue, the messages generated by the same user can be put into the same sub-queue to realize the message merging processing function, and then batch consumption can be realized to improve the processing capability of the consumer end. Furthermore, since the messages generated by the same user are put into the same sub-queue, even if a certain user generates a large number of messages, it will not affect the consumption of messages in other sub-queues, and thus will not affect the real-time performance of business processing of other users.

[0040] In addition, the embodiments of the present disclosure do not need to deploy an independent message queue service, and only need to establish a message table in an existing database. Compared with the middleware method, it can also reduce hardware configuration requirements and maintenance costs.

[0041] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure.

[0042] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0044] Figure 1 A flow chart of a message queue processing method provided by an embodiment of the present disclosure is shown; Figure 2 A schematic diagram of a message distribution process provided by an embodiment of the present disclosure is shown; Figure 3 A schematic diagram of a process of starting a thread provided by an embodiment of the present disclosure is shown; Figure 4 A schematic diagram showing a step-by-step increase in consumption speed provided by an embodiment of the present disclosure is shown; Figure 5A schematic diagram showing a consumption speed step-by-step increase followed by a fuse provided by an embodiment of the present disclosure is shown; Figure 6 A schematic diagram showing a consumption speed provided by an embodiment of the present disclosure, in which a consumption speed increases in a step-by-step manner and then decreases in a step-by-step manner; Figure 7 A schematic diagram of the structure of a message queue processing device provided by an embodiment of the present disclosure is shown; Figure 8 A schematic diagram showing the structure of another message queue processing device provided by an embodiment of the present disclosure is shown; Fig. 9 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0046] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0047] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0048] As the business volume grows, more and more businesses, including some with real-time requirements, are connected to the message queue (MQ) processing mechanism. In the multi-tenant (also known as user) business development scenario, when an order or a piece of data changes, a message notification will be issued, and other dependent data will be updated when the message is consumed. In this scenario, there is no consumption order requirement for messages of different orders from different tenants, but messages of the same order from the same tenant need to be consumed in order, otherwise it will lead to data confusion.

[0049] Research has found that the traditional message consumption model relies on MQ message middleware for implementation, where the producer program produces messages and puts them into a specified queue, and then the consumer program pulls messages from the specified queue for consumption. However, the production environment of this consumption model requires the installation and configuration of the corresponding MQ middleware product, which leads to higher requirements for software and hardware environment, procurement costs, compatibility, etc. In addition, when a user's concurrent message push volume is large, it is easy to cause message congestion, resulting in untimely message consumption by other users, which in turn affects the real-time nature of other users' business processing.

[0050] Based on the above research, the present disclosure provides a message queue processing method, firstly, at least one message table is created in the database, each message table corresponds to a target queue, and the message table is used to store the description information of the message; the message table includes at least one data linked list, and the data linked list is used to store the message content, and each of the data linked lists corresponds to a subqueue of the target queue; then, at least one target message produced by the producer and the description information carried by each of the target messages are obtained, and the description information of the target message includes the user identifier of the target message; finally, according to the user identifier of each of the target messages, the target messages with the same user identifier are assigned to the same message table.

[0051] In the disclosed embodiment, since at least one message table is created in the database, the message table is used to store the description information of the message, each message table corresponds to a target queue, and the message table includes at least one data linked table, the data linked table is used to store the message content, and each data linked table corresponds to a subqueue of the target queue. In this way, through the two-dimensional queue method, the messages produced by the same user can be placed in the same subqueue to realize the message merging processing function, and then batch consumption can be realized, which is conducive to improving the processing capacity of the consumer end. Furthermore, since the messages produced by the same user are placed in the same subqueue, even if a user generates a large number of messages, it will not affect the consumption of messages in other subqueues, and thus will not affect the real-time performance of other users' business processing.

[0052] In addition, this method does not require the deployment of an independent message queue service, but only requires the establishment of a message table in an existing database. Compared with the middleware method, it can also reduce hardware configuration requirements and maintenance costs.

[0053] To facilitate understanding of this embodiment, a message queue processing method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the message queue processing method provided in the embodiment of the present disclosure is generally an electronic device, and the electronic device can be a server, which can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data and artificial intelligence platforms. In other embodiments, the electronic device can also be a terminal device, wherein the terminal device can be a mobile device, a user terminal, a terminal, a handheld device, a computing device, a vehicle-mounted device, a wearable device, and the like.

[0054] In other implementations, the method can also be applied to an implementation environment consisting of a terminal device and a server. In addition, the message queue processing method can also be implemented by a processor calling a computer-readable instruction stored in a memory.

[0055] A message queue processing method provided by an embodiment of the present disclosure is described below in conjunction with the accompanying drawings.

[0056] See also Figure 1 As shown, it is a flowchart of a message queue processing method provided by an embodiment of the present disclosure, such as Figure 1 As shown in , the method includes steps S101 to S103, wherein: S101, create at least one message table in a database, each message table corresponds to a target queue, and the message table is used to store description information of the message; the message table includes at least one data linked table, and the data linked table is used to store message content, and each data linked table corresponds to a subqueue of the target queue.

[0057] Specifically, before creating the message table in the database, the code components of the target queue may be obtained first, and the code components of the target queue may be integrated into the producer end and the consumer end respectively.

[0058] The producer / consumer information refers to the producer / consumer address information, port information, instance name, etc. For example, the producer may be a server that sends messages, and the consumer may be a server that processes messages. Of course, the producer and the consumer may also be integrated into the same device, which is not specifically limited here.

[0059] Optionally, when creating at least one message table in the database, the process may include: creating at least one message table in the database according to the business data volume of the producer and the number of consumer threads, so as to improve the performance of the database.

[0060] Specifically, the number of message tables required can be evaluated based on the concurrency and data volume of the producer's business, and then a corresponding number of message tables can be created. The concurrency refers to the number of threads that consume messages, that is, the number of consuming threads. For example, if there is only one message table, the operation of the message table will be more frequent, so the performance of the message table will be relatively low. One message table can be split into multiple message tables, and then the pressure on a single table can be distributed to other tables. Therefore, the number of message tables actually created can be determined and adjusted according to demand, and the embodiments of the present disclosure do not limit this.

[0061] Exemplarily, see Table 1, which is an example of a message table provided in an embodiment of the present disclosure.

[0062] Table 1

[0063] It can be seen from Table 1 above that the message queue can be stored in a database, and multiple fields can be maintained in the message table, each field corresponding to different description information, wherein a data link table for storing specific message content can be maintained in one field.

[0064] S102, obtaining at least one target message produced by a producer and description information carried by each of the target messages, wherein the description information of the target message includes a user identifier of the target message.

[0065] Among them, the target message refers to the message generated by the user (also known as the tenant) on the producer side. The user can be for each enterprise. For example, for each user (merchant) on the e-commerce platform, the target message will be generated during the process of buyer placing an order, payment and after-sales.

[0066] As can be seen from the above table, the description information of the target message may include: message identifier (IdentityDocument, ID), message type, user identifier (ID), data identifier (ID), message category, priority, message dispatch time, data linked list, number of retries, message status, creation time, etc. Among them, the user identifier is used to distinguish messages from different users, and the data identifier is used to distinguish different data.

[0067] In some embodiments, the message type includes a first type, a second type, and a third type, and different types indicate different consumption speeds, wherein the consumption speed of the first type is greater than the consumption speed of the second type, and the consumption speed of the second type is greater than the consumption speed of the third type. Specifically, the first type refers to a type of consumption according to priority, and the second type and the third type refer to types of consumption according to dispatch time. Therefore, optionally, the first type may also be called a fast type, the second type may also be called a slow type, and the third type may also be called an abnormal type.

[0068] S103: According to the user identifier of each target message, the target messages with the same user identifier are allocated to the same message table.

[0069] Specifically, after the target message and the description information of the target message are acquired, the target messages with the same user identifier may be allocated to the same message table.

[0070] Optionally, when allocating target messages with the same user identifier to the same message table according to the user identifier of each target message, the method may include: According to the user identifier of each target message, the target messages with the same user identifier are allocated to the same message table using a consistent hashing algorithm.

[0071] That is to say, before allocating the target message to the message table, a consistent hashing algorithm can be used according to the user ID to ensure that the data of the same user is in the same message table. In this way, the efficiency and accuracy of allocation can be improved.

[0072] Exemplarily, when allocating target messages with the same user identifier to the same message table according to the user identifier of each target message, the method may include: For each target message, determine whether the target message already exists in the same message table based on the data identifier of the target message; if so, add the target message to the data linked list corresponding to the existing target message; if not, assign the target message to a new data linked list in the same message table.

[0073] It can be understood that when the target message is assigned to the message table, the type of the target message is defaulted to the first type. Figure 2As shown in the figure, after obtaining the target message, the target message is set to the first type, and then the target message can be checked first according to the data identifier of the target message to see if the target message already exists. If it does, the data of the target message is appended to the end of the linked list of the data linked list (data_list) field of the target message; if it does not exist, the target message is added. In this way, it can avoid blocking all message queues due to a large number of messages produced by a certain user, and it is also convenient to perform flow control when consuming messages.

[0074] In some embodiments, after at least one message table is created, the at least one message table can be searched from the database and multiple consumer threads can be started to consume the target messages in the at least one message table. The search refers to calling the database built-in interface to obtain all message tables. The multiple consumer threads include a first type of consumer thread, a second type of consumer thread, and a third type of consumer thread. The first type of consumer thread is used to consume the first type of target message, the second type of consumer thread is used to consume the second type of target message, and the third type of consumer thread is used to consume the third type of target message. In this way, different types of messages are consumed by different consumer threads, which can improve the efficiency of message consumption and avoid message congestion.

[0075] It can be understood that since the consumption speed of the first type is greater than that of the second type, and the consumption speed of the second type is greater than that of the third type, the number of first type consumer threads needs to be greater than that of the second type consumer threads, and the number of the second type consumer threads needs to be greater than that of the third type consumer threads. In this way, it can not only ensure that different types of messages can be consumed in a timely manner, but also avoid waste of resources.

[0076] It should be noted that the more consumer threads there are, the more frequent the access to the message table will be, which will reduce the performance of the message table. Therefore, when the number of consumer threads is large, the number of message tables can be increased to adjust the performance of the message table. After the message table is created, the consumer thread can be started to actively poll the created message table.

[0077] Exemplarily, as shown in Figure 3, the number of the first type of consumer threads can be N, the number of the second type of consumer threads can be N / 2, and the number of the third type of consumer threads can be N / 4, where N is a positive integer greater than or equal to 4, and N is a multiple of 4. In this way, after searching all message tables from the database, various types of consumer threads can be started to consume different types of messages. For example, the consumer thread will poll all message tables, and then query the message table for a preset number (such as 100) of target messages in the state of being consumed in the specified subqueue (data linked list). After the retrieval is completed, the state of the batch of messages will be set to being consumed. If the setting fails, it will not enter the next stage.

[0078] It should be noted that in addition to starting the consumption thread, a cleaning thread is also started, which is used to clean up the message content in the at least one message table that has been consumed before a preset time period. For example, the cleaning thread is used to cyclically clean up the message data in the message table that has been successfully consumed and created 10 days ago, thereby avoiding the situation where the performance is degraded due to the continuous increase in the amount of data in the message table.

[0079] It can be understood that since the target messages are all of the first type by default when assigned to the message table, the first type refers to the type that is consumed according to priority, and there may be situations in which the message consumption fails during the message consumption process. Therefore, in order to avoid continuous repeated consumption of the same target message and affecting the consumption of other target messages, in some embodiments, the method also includes: for any target message, in the process of consuming the target message, counting the first number of repeated consumptions of the target message under the first type, and when the first number of repeated consumptions reaches a first preset threshold, changing the message type of the target message from the first type to the second type.

[0080] That is, in this embodiment, the message queue supports the ability to dynamically adjust the priority queue. When the first type of target message of each user continues to be unsuccessful in consumption, it means that the target message has lost the real-time significance of the message. Therefore, the type of the target message can be changed to the second type, and the second type is consumed according to the delivery time, so that it can be consumed according to the normal delivery time.

[0081] It can be understood that the reason for a message with a large number of retries is that there is a problem with the consumer end consuming the message, such as an error in an interface, which causes continuous failure. If it is not processed, it will block other messages in the same queue. Therefore, in some embodiments, the method may also include: counting the second number of repeated consumptions of the target message under the second type, and when the second number of repeated consumptions reaches the second preset threshold, changing the message type of the target message from the second type to the third type. That is, if the target message continues to be consumed unsuccessfully under the second type, the message type of the target message can be changed from the second type to the third type according to the number of retries, so that it can be consumed by the third type of consumer thread with a slower consumption speed, which is conducive to improving resource utilization efficiency.

[0082] The sizes of the first preset threshold and the second preset threshold are not specifically limited and can be set according to actual needs.

[0083] In addition, since the first type is consumed in order of priority, if the production speed of high-priority target messages is greater than the consumption speed, low-priority messages will not be consumed, thus causing congestion. Therefore, when the dispatch time of the first type of target message is greater than a certain threshold time, the message type of the target message can be changed from the first type to the second type, and the second type of target messages are dispatched in order of dispatch time, so that the problem of low-priority messages not being consumed will not occur.

[0084] Therefore, in some embodiments, the method may further include: for any target message, if the waiting time for dispatching the target message under the first type is longer than a first time threshold, changing the message type of the target message from the first type to the second type. The first time threshold may be determined according to actual needs and is not specifically limited here.

[0085] In some cases, since multiple users (such as enterprises) may share the same message queue, the amount of messages for a certain user may be too large, affecting the messages of other users in the same queue. In response to this situation, each user can set a maximum consumption speed, that is, different users in the same message table have corresponding maximum consumption speeds. Therefore, in this embodiment, the method may also include: when the consumption speed of any target user in the same message table reaches the corresponding maximum consumption speed, the dispatch time of the target message of the target user is extended, and the number of retries of the target message of the target user is increased by 1. Among them, the consumption speed is used to characterize the number of messages consumed per unit time. Specifically, the duration of the extension of the dispatch time of the target message can be set according to actual needs, such as extending it by 30 seconds, one minute, 2 minutes, etc.

[0086] In some embodiments, after the service is started, in order to prevent the consumption speed from increasing too fast, causing the underlying service to be overwhelmed by excessive pressure, when starting multiple consumption threads to consume the target message in the at least one message table, it can include: starting multiple consumption threads, consuming the target message in the at least one message table in a manner that the consumption speed increases in sequence according to a preset time interval, until the consumption speed reaches a preset consumption speed threshold. That is, during normal consumption, if the consumption speed does not reach the preset consumption speed threshold, the consumption speed will be increased step by step until the consumption speed reaches the preset consumption speed threshold, and then consumption will continue at the preset consumption speed threshold.

[0087] Specifically, see Figure 4 As shown in the figure, the message consumption speed is divided into 10 levels. For example, the maximum speed that can be set is 100tps, which can be divided into 10 levels: 10 tps, 20 tps, 30 tps, 40 tps, 50 tps, 60 tps, 70 tps, 80 tps, 90 tps, 100 tps; the time interval of each level can be 5 minutes. Figure 4 As shown in the figure, the consumption speed increases from the initial 10 tps to 100 tps. Among them, tps (Transaction Per Second) refers to the number of transactions that the system can process per second, which is an important indicator to measure the system's processing capacity.

[0088] Optionally, the method also includes: during the process of consuming the target message in the at least one message table, if a failure of the target consumption service is detected, the consumption speed of the first target message of the message category corresponding to the target consumption service is adjusted from the current consumption speed to a preset minimum consumption speed.

[0089] In addition, when it is detected that the target consumption service returns to normal, the consumption speed of the first target message is adjusted in a step-by-step manner to a maximum consumption speed corresponding to the first target message.

[0090] The target consumption service refers to a service for consuming target messages, such as a target message of "payment request", and the target consumption service can be a financial system call service. The preset minimum consumption speed can also be set according to actual needs, such as 0.

[0091] See also Figure 5As shown, that is, when a service (target consumption service) of a certain type of message that the consumer side relies on fails, the consumption of this type of message during the failure period will fail. If this happens, this type of target message (first target message) does not need to be consumed and can be quickly discharged. Therefore, when this happens, the current limiting ladder can be set to the lowest level (this can be called fuse), and when the target consumption service returns to normal, it will return to the original maximum consumption speed. Furthermore, in the process of returning to the original maximum consumption speed, it can be adjusted to the maximum consumption speed corresponding to the first target message in a step-by-step manner.

[0092] In other embodiments, the method further includes: during the process of consuming the target message in the at least one message table, if it is detected that the target consumption service enters a current limiting state, then the consumption speed of the target message corresponding to the target consumption service is reduced in a step-by-step manner.

[0093] Specifically, when a certain type of message consumption speed is too fast, causing the service (target consumption service) that the consumer side relies on to be limited due to traffic overload, it is necessary to reduce the consumption speed of this type of target message to prevent the target consumption service that processes this type of target message from being overwhelmed and causing failure. Therefore, when this happens, the flow limit level of this type of target message can be reduced by one level, and after the batch of target messages is processed, it can be restored to the original speed. In this way, abnormal scenarios can be automatically processed, reducing manual intervention.

[0094] like Figure 6 As shown, when the ladder is descending, it will not be allowed to descend again within a certain period of time (1 minute) to prevent the rapid decline caused by multiple concurrent requests of threads in a short period of time. Furthermore, the step-by-step time interval increases successively. That is, when the ladder is descending, the time interval of the step increase will be expanded, and the time interval is dynamic and can be dynamically adjusted according to the number of declines in a preset period (such as 60 minutes). In this way, it can be prevented that after the decline, the step increase time interval is reached and then increases again, resulting in ups and downs in a short period of time.

[0095] For the step-by-step rate limiting scenario, it can have the ability to adapt when the underlying business is abnormal or the underlying service is overloaded, thereby preventing the underlying system from crashing due to a large number of messages. In addition, the automatic rate limiting capability also reduces the impact of such target messages on other messages, improving the real-time nature of the messages.

[0096] For example, assuming that the maximum flow limiting speed is maxLimitTps, the step increase time interval is levelAddSecondTime (in seconds), the step decrease time interval is levelDownSecondTime (in seconds), the current step number is currentLevel, the current step flow limiting speed is currentLimitTps, the next step increase time of the current step flow limiting speed is nextLevelAddTime (in milliseconds), the last step flow limiting speed decrease time is lastLevelDownTime (in milliseconds), and the number of step decreases within 60 minutes is levelDownCount.

[0097] When maxLimitTps=100, levelAddSecondTime=60, levelDownSecondTime=60, currentLimitTps=maxLimitTps / 10 currentLevel; When the steps increase: nextLevelAddTime=current time (milliseconds) + levelAddSecondTime 60; currentLevel=currentLevel+1; When the stairs go down: nextLevelAddTime=current time (milliseconds) + levelAddSecondTime 60 levelDownCount; lastLevelDownTime = current time (milliseconds); currentLevel=currentLevel-1; When the fuse is blown: nextLevelAddTime=current time (milliseconds) + levelAddSecondTime 60 levelDownCount; lastLevelDownTime = current time (milliseconds); currentLevel=1.

[0098] The message queue processing method provided by the embodiment of the present disclosure is to create at least one message table in the database, the message table is used to store the description information of the message, each message table corresponds to a target queue, and the message table includes at least one data linked table, the data linked table is used to store the message content, and each data linked table corresponds to a sub-queue of the target queue. In this way, through the two-dimensional queue method, the messages produced by the same user can be placed in the same sub-queue to realize the message merging processing function, and then batch consumption can be realized, which is conducive to improving the processing capacity of the consumer end. Furthermore, since the messages produced by the same user are placed in the same sub-queue, even if a user generates a large number of messages, it will not affect the consumption of messages in other sub-queues, and thus will not affect the real-time performance of other users' business processing.

[0099] In addition, this method does not require the deployment of an independent message queue service, but only requires the establishment of a message table in an existing database. Compared with the middleware method, it can also reduce hardware configuration requirements and maintenance costs.

[0100] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0101] Based on the same inventive concept, the embodiment of the present disclosure also provides a message queue processing device corresponding to the above-mentioned message queue processing method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned message queue processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0102] See also Figure 7 , Figure 7 A schematic diagram of a message queue processing device provided by an embodiment of the present disclosure. Figure 7 As shown in , the message queue processing device 700 provided in the embodiment of the present disclosure includes: The creation module 701 is used to create at least one message table in the database, each message table corresponds to a target queue, and the message table is used to store the description information of the message; the message table includes at least one data linked table, and the data linked table is used to store the message content, and each data linked table corresponds to a subqueue of the target queue; The production module 702 is used to obtain at least one target message produced by the producer and the description information carried by each of the target messages, wherein the description information of the target message includes the user identifier of the target message; The configuration module 703 is used to allocate the target messages with the same user identifier to the same message table according to the user identifier of each target message.

[0103] In an optional implementation manner, the description information of the target message further includes a data identifier, and the configuration module 703 is specifically configured to: For each target message, determining, according to the data identifier of the target message, whether the target message already exists in the same message table; If it exists, then add the target message to the data linked list corresponding to the existing target message; If not, the target message is assigned to a new data linked list in the same message table.

[0104] In an optional implementation manner, the configuration module 703 is specifically used to: According to the user identifier of each target message, the target messages with the same user identifier are allocated to the same message table using a consistent hashing algorithm.

[0105] In an optional implementation, the description information of the target message includes a message type, and the message type includes a first type, a second type, and a third type. Different types indicate different consumption speeds, wherein the consumption speed of the first type is greater than the consumption speed of the second type, and the consumption speed of the second type is greater than the consumption speed of the third type; see Figure 8 As shown, the device also includes: The consumption module 704 is used to search the at least one message table from the database and start multiple consumption threads to consume the target message in the at least one message table. The multiple consumption threads include a first type of consumption thread, a second type of consumption thread and a third type of consumption thread. The first type of consumption thread is used to consume the first type of target message, the second type of consumption thread is used to consume the second type of target message, and the third type of consumption thread is used to consume the third type of target message.

[0106] In an optional implementation, the first type refers to a type of consumption according to priority, and the second type and the third type refer to types of consumption according to dispatch time.

[0107] In an optional implementation, the number of the first-category consumer threads is greater than the number of the second-category consumer threads, and the number of the second-category consumer threads is greater than the number of the third-category consumer threads.

[0108] In an optional implementation manner, the creation module 701 is specifically used to: At least one message table is created in the database according to the business data volume of the producer and the number of consumer threads.

[0109] In an optional implementation, when the target message is assigned to the message table, the type of the target message is defaulted to the first type, and the configuration module 703 is further used to: For any target message, during the process of consuming the target message, the first repeated consumption times of the target message under the first type are counted. When the first repeated consumption times reaches a first preset threshold, the message type of the target message is changed from the first type to the second type.

[0110] In an optional implementation manner, the configuration module 703 is further configured to: The second repeated consumption times of the target message under the second type are counted, and when the second repeated consumption times reaches a second preset threshold, the message type of the target message is changed from the second type to the third type.

[0111] In an optional implementation, when the target message is assigned to the message table, the type of the target message is defaulted to the first type, and the configuration module 703 is further used to: For any target message, if the dispatch waiting time of the target message under the first type is greater than a first time threshold, the message type of the target message is changed from the first type to the second type.

[0112] In an optional implementation manner, different users in the same message table have corresponding maximum consumption speeds respectively, and the configuration module 703 is further used to: When the consumption speed of any target user in the same message table reaches the corresponding maximum consumption speed, the dispatching time of the target message of the target user is extended, and the number of retries of the target message of the target user is increased by 1.

[0113] In an optional implementation manner, the consumption module 704 is specifically used for: A plurality of consumption threads are started to consume target messages in the at least one message table at a consumption speed increasing in sequence according to a preset time interval.

[0114] In an optional implementation manner, the description information of the target message further includes a message category; and the consumption module 704 is further configured to: During the process of consuming the target message in the at least one message table, if a failure of the target consumption service is detected, the consumption speed of the first target message of the message category corresponding to the target consumption service is adjusted from the current consumption speed to a preset minimum consumption speed.

[0115] In an optional implementation manner, the consumption module 704 is further used to: When it is detected that the target consumption service returns to normal, the consumption speed of the first target message is adjusted in a step-by-step manner to a maximum consumption speed corresponding to the first target message.

[0116] In an optional implementation manner, the consumption module 704 is further used to: In the process of consuming the target message in the at least one message table, if it is detected that the target consumption service enters a current limiting state, the consumption speed of the target message corresponding to the target consumption service is reduced in a step-by-step manner.

[0117] In an optional implementation, the step-by-step time intervals increase sequentially.

[0118] For descriptions of the processing flow of each module in the above-mentioned device and the interaction flow between each module, reference can be made to the relevant descriptions in the above-mentioned method embodiment, which will not be described in detail here.

[0119] Corresponding to the above-mentioned message queue processing method, the embodiment of the present disclosure further provides an electronic device 900, such as Fig. 9 FIG. 1 is a schematic diagram of the structure of an electronic device 900 provided in an embodiment of the present disclosure, including: Processor 901, memory 902, and bus 903. The memory 902 is used to store execution instructions, including internal memory 9021 and external memory 9022; the internal memory 9021 is also called internal memory, which is used to temporarily store the operation data in the processor 901 and the data exchanged with the external memory 9022 such as a hard disk. The processor 901 exchanges data with the external memory 9022 through the internal memory 9021.

[0120] In the embodiment of the present application, the memory 902 is specifically used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 901. That is, when the electronic device 900 is running, the processor 901 communicates with the memory 902 through the bus 903, or the processor 901 communicates with the memory 902 in other ways, so that the processor 901 executes the application code stored in the memory 902, and then executes the steps of the message queue processing method described in any of the above embodiments.

[0121] Among them, the memory 902 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0122] Processor 901 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0123] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 900. In other embodiments of the present application, the electronic device 900 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0124] The embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any one of the message queue processing methods described in the above method embodiments is executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0125] In addition, an embodiment of the present disclosure also provides a computer program product, which stores a computer program. When the computer program is executed by a processor, the steps of the message queue processing method provided by any of the above embodiments of the present disclosure are executed. For details, please refer to the above method embodiments, which will not be repeated here.

[0126] The computer program product may be implemented in hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, which may be a volatile or non-volatile computer-readable storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0127] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the above-described equipment and devices can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0130] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0131] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A message queue processing method, characterized in that: include: Creating at least one message table in a database, each message table corresponding to a target queue, and the message table is used to store description information of a message; The message table includes at least one data linked list, the data linked list is used to store message content, and each of the data linked lists corresponds to a sub-queue of the target queue; Acquire at least one target message produced by a producer and description information carried by each of the target messages, wherein the description information of the target message includes a user identifier of the target message; According to the user identification of each target message, the target messages with the same user identification are allocated to the same message table.

2. The method according to claim 1, characterized in that The description information of the target message also includes a data identifier, and the target messages with the same user identifier are allocated to the same message table according to the user identifier of each target message, including: For each target message, determining, according to the data identifier of the target message, whether the target message already exists in the same message table; If it exists, then add the target message to the data linked list corresponding to the existing target message; If not, the target message is assigned to a new data linked list in the same message table.

3. The method according to claim 1, characterized in that The step of allocating target messages with the same user identifier to the same message table according to the user identifier of each target message includes: According to the user identifier of each target message, the target messages with the same user identifier are allocated to the same message table using a consistent hashing algorithm.

4. The method according to claim 1, characterized in that: The description information of the target message includes a message type, and the message type includes a first type, a second type, and a third type. Different types indicate different consumption speeds, wherein the consumption speed of the first type is greater than the consumption speed of the second type, and the consumption speed of the second type is greater than the consumption speed of the third type; the method further includes: The at least one message table is searched from the database, and multiple consumer threads are started to consume the target messages in the at least one message table, the multiple consumer threads include a first type of consumer thread, a second type of consumer thread and a third type of consumer thread, the first type of consumer thread is used to consume the first type of target message, the second type of consumer thread is used to consume the second type of target message, and the third type of consumer thread is used to consume the third type of target message.

5. The method according to claim 4, characterized in that The first type refers to a type of consumption according to priority, and the second type and the third type refer to types of consumption according to distribution time.

6. The method according to claim 4, characterized in that The number of the first type of consumer threads is greater than the number of the second type of consumer threads, and the number of the second type of consumer threads is greater than the number of the third type of consumer threads.

7. The method according to claim 1, characterized in that The step of creating at least one message table in a database comprises: At least one message table is created in the database according to the business data volume of the producer and the number of consumer threads.

8. The method according to claim 4, characterized in that When allocating the target message to the message table, the type of the target message is defaulted to the first type, and the method further includes: For any target message, during the process of consuming the target message, the first repeated consumption times of the target message under the first type are counted. When the first repeated consumption times reaches a first preset threshold, the message type of the target message is changed from the first type to the second type.

9. The method according to claim 8, characterized in that The method further comprises: The second repeated consumption times of the target message under the second type are counted, and when the second repeated consumption times reaches a second preset threshold, the message type of the target message is changed from the second type to the third type.

10. The method according to claim 4, characterized in that When allocating the target message to the message table, the type of the target message is defaulted to the first type, and the method further includes: For any target message, if the dispatch waiting time of the target message under the first type is greater than a first time threshold, the message type of the target message is changed from the first type to the second type.

11. The method according to claim 4, characterized in that Different users in the same message table have corresponding maximum consumption speeds respectively. The method further includes: When the consumption speed of any target user in the same message table reaches the corresponding maximum consumption speed, the dispatching time of the target message of the target user is extended, and the number of retries of the target message of the target user is increased by 1.

12. The method according to claim 4, characterized in that The starting of multiple consumer threads to consume the target message in the at least one message table includes: A plurality of consumption threads are started to consume target messages in the at least one message table at a consumption speed increasing in sequence according to a preset time interval.

13. The method according to claim 12, characterized in that The description information of the target message also includes a message category; the method further includes: During the process of consuming the target message in the at least one message table, if a failure of the target consumption service is detected, the consumption speed of the first target message of the message category corresponding to the target consumption service is adjusted from the current consumption speed to a preset minimum consumption speed.

14. The method according to claim 13, characterized in that The method further comprises: When it is detected that the target consumption service returns to normal, the consumption speed of the first target message is adjusted in a step-by-step manner to a maximum consumption speed corresponding to the first target message.

15. The method according to claim 12, characterized in that The method further comprises: In the process of consuming the target message in the at least one message table, if it is detected that the target consumption service enters a current limiting state, the consumption speed of the target message corresponding to the target consumption service is reduced in a step-by-step manner.

16. The method according to claim 15, characterized in that The step-wise time intervals increase sequentially.

17. A message queue processing device, characterized in that: include: A creation module, used to create at least one message table in the database, each message table corresponds to a target queue, and the message table is used to store description information of the message; The message table includes at least one data linked list, the data linked list is used to store message content, and each of the data linked lists corresponds to a sub-queue of the target queue; A production module, used for acquiring at least one target message produced by a producer and each target message carries description information of the target message, wherein the description information of the target message includes a user identifier of the target message; The configuration module is used to allocate the target messages with the same user identification to the same message table according to the user identification of each target message.

18. An electronic device, characterized in that: It includes a processor, a memory and a bus, the memory stores machine-readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the message queue processing method according to any one of claims 1 to 16 is performed.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the message queue processing method according to any one of claims 1 to 16 is executed.

Images