Cloud Native-Based Message Processing Method, Device, Equipment and Storage Medium

Abstract

The present invention discloses a message processing method, device, equipment and storage medium based on cloud native. The method includes: receiving a message processing request, where the message processing request includes a target operation type; listening to the current message status of the current message stream abstraction; when the current message status is a fault-free status, executing a target operation process corresponding to the target operation type to process the message processing request; when the current message status is a faulty status, executing a message recovery process, and when it is monitored that the current message status is a fault-free status, executing a target operation process corresponding to the target operation type to process the message processing request. The method will only execute the target operation process corresponding to the target operation type to process the message processing request when ensuring that the current message status is a fault-free status, which helps to ensure the consistency of subsequent message processing.

Description

Technical Field

[0001] The present invention relates to the field of cloud native technologies, and in particular, to a message processing method, apparatus, device, and storage medium based on cloud native. Background Art

[0002] In the existing message processing system, after a message processing fails once, when the message processing is performed again, the consistency of message reading and writing cannot be guaranteed. Summary of the Invention

[0003] Embodiments of the present invention provide a message processing method, apparatus, computer device, and storage medium based on cloud native to solve the problem that the existing message processing system cannot guarantee the consistency of message reading and writing.

[0004] A message processing method based on cloud native includes:

[0005] Receiving a message processing request, where the message processing request includes a target operation type;

[0006] Listening to the current message state of the current message flow abstraction;

[0007] When the current message state is a fault-free state, executing a target operation process corresponding to the target operation type to process the message processing request;

[0008] When the current message state is a faulty state, executing a message recovery process, and when it is monitored that the current message state is a fault-free state, executing a target operation process corresponding to the target operation type to process the message processing request.

[0009] A message processing apparatus based on cloud native includes:

[0010] A processing request receiving module, configured to receive a message processing request, where the message processing request includes a target operation type;

[0011] A message state listening module, configured to listen to the current message state of the current message flow abstraction;

[0012] A first processing module, configured to execute a target operation process corresponding to the target operation type to process the message processing request when the current message state is a fault-free state;

[0013] A second processing module, configured to execute a message recovery process when the current message state is a faulty state, and execute a target operation process corresponding to the target operation type to process the message processing request when it is monitored that the current message state is a fault-free state.

[0014] A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the above-mentioned cloud-native based message processing method is implemented.

[0015] A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the above-mentioned cloud-native based message processing method is implemented.

[0016] For the above-mentioned cloud-native based message processing method, apparatus, computer device and storage medium, when the current Broker receives a message processing request, it needs to first monitor the current message state abstracted from the current message stream. Only when it is ensured that the current message state is a fault-free state, will it execute the target operation process corresponding to the target operation type to process the message processing request, which helps to ensure the consistency of subsequent message processing. Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a schematic diagram of an application environment of the cloud-native based message processing method in an embodiment of the present invention;

[0019] Figure 2 It is a flowchart of the cloud-native based message processing method in an embodiment of the present invention;

[0020] Figure 3 It is another flowchart of the cloud-native based message processing method in an embodiment of the present invention;

[0021] Figure 4 It is another flowchart of the cloud-native based message processing method in an embodiment of the present invention;

[0022] Figure 5 It is another flowchart of the cloud-native based message processing method in an embodiment of the present invention;

[0023] Figure 6 It is another flowchart of the cloud-native based message processing method in an embodiment of the present invention;

[0024] Figure 7 It is another flowchart of the cloud-native based message processing method in an embodiment of the present invention;

[0025] Figure 8 It is another flowchart of the cloud-native based message processing method in an embodiment of the present invention;

[0026] Figure 9 It is a schematic diagram of the cloud-native based message processing device in an embodiment of the present invention;

[0027] Figure 10 It is a schematic diagram of a computer device in an embodiment of the present invention. Detailed implementation manners

[0028] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0029] The cloud-native based message processing method provided by the embodiments of the present invention can be applied in the application environment as Figure 1 shown. Specifically, the cloud-native based message processing method is applied in a cloud-native based message processing middleware architecture system, which includes at least one Pulsar cluster. Each Pulsar cluster includes a Broker cluster, a Bookeeper cluster, and a Zookeeper cluster. Each Broker cluster includes at least one Broker (message service), and each Bookeeper cluster includes at least one Bookie (message storage).

[0030] Each Broker (message service) includes a dispatcher, a load balancer connected to the dispatcher, global replicators, a managed ledger, and a cache. The managed ledger is connected to the Bookie (message storage) through a BK Client (BZ interface).

[0031] Each Bookie (message storage) is used to implement message storage.

[0032] The Zookeeper cluster is connected to each Broker (message service) and Bookie (message storage), and interacts with the Zookeeper clusters (ZK) of other Pulsar clusters.

[0033] In one embodiment, as Figure 2As shown, a cloud-native based message processing method is provided. Taking the cloud-native based message processing middleware architecture system in Figure 1 as an example for illustration, the current Broker includes the following steps:

[0034] S201: Receive a message processing request, where the message processing request includes a target operation type;

[0035] S202: Listen to the current message state of the current message stream abstraction;

[0036] S203: When the current message state is a fault-free state, execute a target operation process corresponding to the target operation type to process the message processing request;

[0037] S204: When the current message state is a faulty state, execute a message recovery process. When it is monitored that the current message state is a fault-free state, execute a target operation process corresponding to the target operation type to process the message processing request.

[0038] The current Broker refers to the Broker that receives the message processing request at the current moment of the system.

[0039] Among them, the message processing request refers to the request received by the system at the current moment for processing messages. The message processing request includes, but is not limited to, a message write request, a message read request, and a message publish request. The target operation type refers to the type used to process the message as defined in the message processing request. The target operation type includes, but is not limited to, a write operation type, a read operation type, and a publish operation type.

[0040] As an example, in step S201, the current Broker can receive a message processing request sent by a user through a client. The message processing request can be a message write request or a message publish request sent by a message producer through the client. The target operation type corresponding to the message write request is a write operation type, and the target operation type corresponding to the message publish request is a publish operation type; it can also be a message read request sent by a message consumer through the client, and the target operation type corresponding to the message read request is a read operation type.

[0041] Among them, the current message flow abstraction refers to the message flow abstraction (managed ledger) in the current Broker. The current message state is a state used to reflect whether the current message flow abstraction in the system can work properly at the current moment. Generally speaking, a Pulsar cluster contains multiple Brokers, each of which is stateless, while the message flow abstraction (managed ledger) is a module used to implement state management. That is, when writing messages, the writer process continuously adds messages to the end of the flow; when a message consumer reads or consumes a certain message, the consumption position of each message consumer is determined to achieve state management.

[0042] The message flow abstraction (managed ledger) is an append-only data structure, and there is only one writer process. This writer process is responsible for writing to multiple Bookeeper storage nodes (i.e., Bookies), and the entries of the message flow abstraction (managed ledger) will be replicated to multiple Bookies. Ledgers themselves have very simple semantics: A Broker can create a message flow abstraction (managed ledger), add content to the ledger, and close the ledger; when a ledger is closed, unless explicitly writing data or the ledger is closed due to the writer process crashing, this ledger will only be opened in read-only mode; when the entries in the ledger are no longer useful, the entire ledger can be deleted.

[0043] As an example, in step S202, when the current Broker receives a message processing request, it needs to monitor the current message state in the current message flow abstraction to evaluate whether normal message writing, reading, or publishing operations can be performed.

[0044] As an example, in step S203, when the current Broker monitors that the current message state of the current message flow abstraction is a fault-free state, it indicates that the current message flow abstraction in the current Broker can work properly. At this time, the target operation process corresponding to the target operation type can be executed to process the message processing request.

[0045] As an example, in step S204, when the current Broker monitors that the current message state of the current message stream abstraction is in a faulty state, it indicates that the current message stream abstraction in the current Broker cannot work properly. If the target operation process corresponding to the target operation type is directly executed to process the message processing request, it is very likely that there will be inconsistent situations in message processing. At this time, the message recovery process needs to be executed first for message recovery processing. For example, the message recovery process can be used to perform a rollback operation to roll back the system state to before the fault occurred; after the message recovery processing is executed, the current message state of the current message stream abstraction is monitored again. Only when the current message state is in a fault-free state, the target operation process corresponding to the target operation type is executed to process the message processing request, which helps to ensure the consistency of message processing. It can be understood that since the current message stream abstraction can only be written by one writer process, there will be no conflict during the write operation of this writer process, and the write operation is relatively efficient. When its current message state is in a faulty state, the current message stream abstraction will start the message recovery process for recovery processing and re-determine its current message state. Only when the current message state is in a fault-free state, the target operation process corresponding to the target operation type will be executed to process the message processing request to ensure the consistency of subsequent message processing.

[0046] In this example, executing the target operation process corresponding to the target operation type to process the message processing request specifically includes: (1) If the target operation type is a write operation type, execute the write operation process, obtain the message to be processed from the message processing request, and write the message to be processed into the current Bookie. Specifically, through the message stream abstraction (managed ledger), write the message to be processed into the current Bookie, which helps to ensure the writing efficiency of the message to be processed. (2) If the target operation type is a read operation type, execute the write operation process to pull the required message to be processed from the current Bookie. In this example, the current Broker can use the stream pull method to pull the required message to be processed from the current Bookie to improve the acquisition efficiency of the message to be processed. The stream pull is an improved version of long polling. It not only achieves zero waiting between individual calls and requests, but also provides a two-way message stream. Using the stream pull method, the end-to-end latency can be lower than that of all existing long polling message systems (such as Kafka). (3) If the target operation type is a publish operation type, execute the publish operation process to publish the message to be processed corresponding to the message processing request.

[0047] In the message processing method based on cloud native provided by this embodiment, when the current Broker receives a message processing request, it needs to first monitor the current message state abstracted from the current message flow. Only when it is ensured that the current message state is a fault-free state, will it execute the target operation process corresponding to the target operation type to process the message processing request, which helps to ensure the consistency of subsequent message processing.

[0048] In one embodiment, as Figure 3 shown, step S301, that is, executing the target operation process corresponding to the target operation type to process the message processing request, includes:

[0049] S301: When the target operation type is a write operation type, obtain the message to be processed and the message processing type from the message processing request;

[0050] S302: When the message processing type is a batch processing type, obtain the current batch identifier and the message sequence identifier corresponding to the message to be processed, and determine whether there is a batch index status queue corresponding to the current batch identifier;

[0051] S303: If there is a batch index status queue, write the message to be processed into the current Bookie, and update the current index status corresponding to the message sequence identifier in the batch index status queue;

[0052] S304: If there is no batch index status queue, write the message to be processed into the current Bookie, create a batch index status queue corresponding to the current batch identifier, and record the current index status corresponding to the message sequence identifier;

[0053] S305: Monitor the current index status of the batch index status queue, and release the storage space corresponding to the batch index status queue when the current index status is the confirmed completion status.

[0054] Among them, the message to be processed refers to the message to be processed in this message processing request. Understandably, when the target operation type is a write operation type, that is, when the message processing request is a message write request, the message to be processed refers to the message carried in the message processing request. The message processing type refers to the type of processing that needs to be performed on the message to be processed this time.

[0055] As an example, in step S301, when the current Broker monitors that the current message state is a fault-free state and the target operation type is a write operation type, it needs to parse the message processing request, and obtain the message to be processed and the message processing type from the message processing request, so as to perform a message write operation according to the message to be processed and the message processing type.

[0056] Among them, the batch processing type refers to the processing type that requires batch writing of multiple messages. The batch index status queue refers to a pre-created queue for storing the progress status of batch writing of multiple messages.

[0057] As an example, in step S302, when the current Broker identifies that the message processing type in the message processing request is the batch processing type, it indicates that the user performs batch message writing through the client. At this time, it is necessary to read the current batch identifier and the message sequence identifier in the message processing request. The current batch identifier is used to uniquely identify which specific batch of messages it is, and the message sequence identifier is used to uniquely identify each message in the batch of messages to be processed. In this example, after the current Broker obtains the current batch identifier and the message sequence identifier from the messages to be processed, it can query whether there is a batch index status queue corresponding to the current batch identifier to determine whether the same batch of messages has been written into the Bookie before the current moment of the system.

[0058] As an example, in step S303, when there is a batch index status queue for the current Broker, it indicates that before the current moment of the system, the same batch of messages has been written into the current Bookie. At this time, the current Broker writes the messages to be processed into the current Bookie and updates the current index status corresponding to the message sequence identifier in the batch index status queue, that is, updates the current index status corresponding to the message sequence identifier in the batch index status queue from the to-be-transmitted status to the confirmed completion status. It can be understood that the message processing request may also carry the number of message batches corresponding to the current batch identifier. The number of message batches can be understood as the number of times of batch transmission required. For example, when a batch of message A needs to be written into the current Bookie, it can be divided into 10 batches for writing, then the number of message batches is 10, and each batch of messages to be processed carries its unique message sequence identifier A1... A10, etc. The current index status corresponding to each message sequence identifier recorded in the batch index status queue can reflect the status of whether the messages to be processed have completed the writing operation.

[0059] As an example, in step S304, when there is a batch index status queue for the current Broker, it indicates that before the current moment of the system, the same batch of messages has not been written into the current Bookie. At this time, the current Broker writes the messages to be processed into the current Bookie, and then needs to create a batch index status queue corresponding to the current batch identifier. In the created batch index status queue, record the current index status corresponding to the message sequence identifier, that is, record the current index status corresponding to the message sequence identifier in the batch index status queue as the confirmed completion status.

[0060] As an example, in step S305, the current Broker listens to the current index status in the batch index status queue. When all current index statuses are in the confirmed completion status, it is determined that all messages in the same batch corresponding to the current batch identifier have been written to the current Bookie. At this time, the storage space corresponding to the batch index status queue is released.

[0061] Understandably, when the target operation type is a write operation type and the message processing type is a batch processing type, using the batch index status queue to monitor the progress of writing all pending messages in the same batch to the current Bookie helps ensure that all pending messages in the same batch can be accurately written to the current Bookie.

[0062] In one embodiment, as Figure 4 shown, step S301, that is, executing the target operation process corresponding to the target operation type to process the message processing request, includes:

[0063] S401: When the target operation type is a write operation type, obtain the pending message and the message processing type from the message processing request;

[0064] S402: When the message processing type is a sharding processing type, obtain multiple sharded messages corresponding to the pending message;

[0065] S403: Store the multiple sharded messages in the current cache;

[0066] S404: Write all the sharded messages in the current cache to the current Bookie, and update the message transmission status of each sharded message in the current cache;

[0067] S405: If all message transmission statuses are in the sharding completion status, release the current cache.

[0068] Among them, the pending message refers to the message to be processed in this message processing request. Understandably, when the target operation type is a write operation type, that is, when the message processing request is a message write request, the pending message refers to the message carried in the message processing request. The message processing type refers to the type of processing that needs to be performed on the pending message this time.

[0069] As an example, in step S401, when the current Broker monitors that the current message status is in a fault-free status and the target operation type is a write operation type, it needs to parse the message processing request and obtain the pending message and the message processing type from the message processing request, so as to perform a message write operation according to the pending message and the message processing type.

[0070] Among them, the sharding processing type refers to dividing the same data into multiple sharded messages, so that the storage space required for each sharded message is smaller, which helps to improve the resource utilization rate of the messages to be processed for storage.

[0071] As an example, in step S402, when the current Broker recognizes that the message processing type in the message processing request is the sharding processing type, it can perform sharding processing on the message to be processed based on a preset sharded message capacity or a user-defined sharded message capacity, and obtain multiple sharded messages.

[0072] The current cache refers to the cache (Cache) in the current Broker, which is used to cache the messages that need to be processed.

[0073] As an example, in step S403, after the current Broker divides the message to be processed into multiple sharded messages, it needs to store these sharded messages in the current cache in the current Broker, which helps to ensure the security of the transmission of the message to be processed and avoid the loss of some sharded messages during the process of writing the sharded messages into the Bookie, thus affecting the integrity of the entire message to be processed.

[0074] As an example, in step S404, when the current Broker caches all the sharded messages in the current cache, it can write all the sharded messages in the current cache into the current Bookie. The current Bookie can be one Bookie or multiple Bookies to implement the write operation on the sharded messages. After each sharded message is written into the current Bookie, it is necessary to update the message transmission status of each sharded message in the current cache, that is, update the message transmission status corresponding to the sharded message from the to-be-transmitted status to the sharded-completed status.

[0075] As an example, in step S405, when the current Broker monitors that the message transmission status corresponding to all the sharded messages is the sharded-completed status, it is determined that all the messages to be transmitted have been written into the current Bookie. At this time, it is necessary to release the current cache so that the current cache can perform cache processing for other messages, which helps to ensure the resource utilization rate.

[0076] In this example, when the message producer publishes or writes a message to be processed, if the size of the message to be processed is greater than the maximum allowed publish payload size, the message to be processed needs to be split into multiple shard messages, and the multiple shard messages are published to the current Broker individually or sequentially. The current Broker stores all the received shard messages in the current cache, stores the shard messages and other messages to be processed on the message flow abstraction (managed ledger) in the same way, and the message flow abstraction (managed ledger) writes all the shard messages and other messages to be processed to the current Bookie.

[0077] Understandably, when the message producer splits the message to be processed into multiple shard messages and stores them in at least one current Bookie, when the message consumer reads the message to be processed, that is, when the target operation type of the message processing request sent to the current Broker is the read operation type and the message processing type is the shard processing type, a buffer for message merging needs to be created in the current cache so that after all the shard messages are collected, they can be merged into the message to be processed. Generally speaking, when the message producer fails to publish all the shard messages corresponding to the message to be processed and the message consumer cannot receive all the shard messages corresponding to the same message to be processed within the target expiration time, it is determined that the publication of the message to be processed fails. At this time, the shard messages in the buffer can be cleared to save storage space and improve resource utilization. The target expiration time can be the system default expiration time or a user-defined expiration time, such as 1h.

[0078] In one embodiment, as Figure 5 shown, the cloud-native based message processing method further includes:

[0079] S501: Monitor the message storage duration corresponding to the message to be processed in the current Bookie;

[0080] S502: If the message storage duration is greater than the target duration threshold, execute the data migration process to migrate the message to be processed in the current Bookie to the disk space.

[0081] Among them, the message storage duration refers to the duration for which the message to be processed is stored in the current Bookie. The target duration threshold is a threshold set in advance for evaluating whether the message storage duration reaches a relatively long standard. The data migration process is a process for implementing data migration.

[0082] As an example, in step S501, the system monitors in real time the message storage duration corresponding to each message to be processed in the current Bookie. The message storage duration is the time difference between the current moment of the system and the message writing moment corresponding to the message to be processed. The message writing moment corresponding to the message to be processed refers to the time when the message to be processed is written into the current Bookie.

[0083] As an example, in step S502, when it is monitored that the message storage duration corresponding to the message to be processed is greater than the target duration threshold, it indicates that the message to be processed has been stored in the current Bookie for a long time and is very likely to be old data with low timeliness. At this time, the message to be processed with a large message storage duration can be migrated from the current Bookie to a disk space with lower cost to release the current Bookie with higher cost, which helps to reduce the message storage cost.

[0084] It can be understood that storing the message to be processed with a message storage duration greater than the target duration threshold in a disk space with lower cost can not only reduce the storage cost, but also naturally isolate the new and old data, avoid subsequent conflicts and contentions in IO reading and writing, and there will be no large-scale copy and recovery operations during processes such as node expansion and error recovery, which helps to improve the high availability of the system.

[0085] In one embodiment, as Figure 6 shown, the message processing method based on cloud native further includes:

[0086] S601: Receive a concurrent storage request formed based on the message to be processed;

[0087] S602: According to the concurrent storage request, call a concurrent storage process to concurrently write the message to be processed into the associated Bookies corresponding to the first quantity;

[0088] S603: Listen to the storage feedback messages returned by the associated Bookies and count the second quantity of the storage feedback messages;

[0089] S604: Obtain a message feedback ratio according to the first quantity and the second quantity;

[0090] S605: If the message feedback ratio is greater than the target ratio threshold, control the concurrent storage process to stop working.

[0091] Among them, the concurrent storage request is a request for implementing concurrent storage of messages. The concurrent storage process is a pre-configured process for implementing concurrent storage. The associated Bookies are other Bookies except the current Bookie that already stores the message to be processed. The target ratio threshold is a threshold set in advance for evaluating whether the ratio reaches a large standard.

[0092] As an example, in step S601, the current Broker can receive a concurrent storage request for a pending message, so as to perform concurrent storage on the pending message according to the concurrent storage request to ensure the security of the pending message.

[0093] As an example, in step S602, the current Broker can, according to the concurrent storage request, call a pre-set concurrent storage process, and through the Global Replicators in the current Broker, concurrently write the pending message into the associated Bookies corresponding to the first quantity. Here, the first quantity is the quantity for concurrent processing of the pending message, that is, the quantity of the associated Bookies to which the writing is performed.

[0094] As an example, in step S603, the current Broker can monitor in real time the storage feedback messages returned by the associated Bookies. The storage feedback messages can reflect that the pending message has been replicated and stored in the associated Bookies. In this example, the current Broker will count in real time the quantity of the received storage feedback messages and determine it as the second quantity. The second quantity can reflect how many associated Bookies have stored the pending message at the current moment of the system.

[0095] As an example, in step S604, the current Broker can, according to the first quantity corresponding to the sending of the pending message and the second quantity corresponding to the receiving of the storage feedback messages, determine the quotient of the second quantity and the first quantity as the message feedback ratio.

[0096] As an example, in step S605, the current Broker can calculate the message feedback ratio in real time, compare the message feedback ratio with a pre-set target ratio threshold in the system. When the message feedback ratio is greater than the target ratio threshold, it is considered that replicating and storing the pending message in more associated Bookies can effectively ensure the security of the pending message. At this time, in order to save system resources, the concurrent storage process can be controlled to stop working.

[0097] In this example, Pulsar adopts a protocol similar to the Qurom protocol to provide an available Bookie pool (i.e., associated Bookies). When executing a concurrent storage process, the messages to be processed can be concurrently written to a part of the associated Bookies. As long as partial success is returned, it is considered that the replicated storage is successful. For example, when the message feedback ratio determined by the first quantity and the second quantity is greater than the target ratio threshold (e.g., 50%), it is considered that the replicated storage of the messages to be processed is completed. By using this concurrent storage method, data replication can be carried out more efficiently, especially when there are many data replicas, and the effect is more significant. It can be understood that the replicated storage of each message to be processed is only stored in different Bookies and does not go through different Brokers for storage processing, which can simplify the storage processing flow.

[0098] In one embodiment, as Figure 7 shown, step S301, that is, execute the target operation process corresponding to the target operation type to process the message processing request, including:

[0099] S701: When the target operation type is the publish operation type, obtain the message to be processed, the message topic, and the current producer from the message processing request;

[0100] S702: Based on the message topic, determine the producer mode corresponding to the message topic;

[0101] S703: If the producer mode is not the producer exclusive mode, directly publish the message to be processed;

[0102] S704: If the producer mode is the producer exclusive mode, obtain the target permission corresponding to the message topic and the current permission of the current producer, perform permission verification based on the target permission and the current permission, and publish the message to be processed after the permission verification passes.

[0103] Among them, the publish operation type is the type used to implement the message publish operation. The message topic refers to the topic corresponding to the message to be processed. The current producer refers to the producer that triggers the message processing request. The producer exclusive mode refers to the mode in which only one producer can exclusively use a certain message topic.

[0104] As an example, in step S701, when the current Broker monitors that the current message status is the fault-free status and the target operation type is the publish operation type, it needs to parse the message processing request and obtain information such as the message to be processed, the message topic, and the current producer from the message processing request.

[0105] As an example, in step S702, after the current Broker obtains the message topic from the message processing request, it is necessary to query the records in the system database according to the message topic to determine the producer mode corresponding to the message topic, so as to evaluate whether the producer mode is the producer exclusive mode, and then determine whether the current producer has the permission to publish the pending message corresponding to the message topic.

[0106] As an example, in step S703, when the producer mode corresponding to the message topic is not the producer exclusive mode, it means that the publication of the pending message corresponding to the message topic is not restricted by the producer identity. At this time, the pending message can be directly published.

[0107] Among them, the target permission corresponding to the message topic refers to the permission to publish the message corresponding to the message topic. The current permission of the current producer is the permission of the messages that the current producer can publish.

[0108] As an example, in step S704, when the producer mode corresponding to the message topic is the producer exclusive mode, it means that the publication of the pending message corresponding to the message topic is restricted by the producer identity. It is necessary to obtain the target permission corresponding to the message topic and the current permission of the current producer; perform permission verification according to the target permission and the current permission. Only after the permission verification passes will the pending message be published, which can manage the message publication process for different message topics to meet user needs.

[0109] In this example, in step S704, if the producer mode is the producer exclusive mode, the target permission corresponding to the message topic and the current permission of the current producer are obtained, and permission verification is performed based on the target permission and the current permission. After the permission verification passes, the message to be processed is published. Specifically, it includes: (1) If the producer mode is the producer exclusive mode, it is determined whether there is an existing producer for the message topic. The existing producer refers to a producer who has sent other messages of the same message topic before the current moment of the system. (2) If there is no existing producer for the message topic, the current producer is associated with the message topic, and the message to be processed is published. That is, when there is no existing producer for the message topic, it means that although the producer mode of a certain message topic is the producer exclusive mode, no existing producer has published the message corresponding to the message topic before the current moment of the system. At this time, the current producer and the message topic can be associated, and the message to be processed is published. (3) If there is an existing producer for the message topic, permission verification is performed on the current producer and the existing producer. After the permission verification passes, the message to be processed is published. In this example, if the current permission of the current producer is greater than the target permission corresponding to the message topic, it means that the current producer has the permission to publish the message to be processed corresponding to the message topic, and it is determined that the permission verification passes, and the message to be processed can be published; if the current permission of the current producer is not greater than the target permission corresponding to the message topic, it means that the current producer does not have the permission to send the message to be processed corresponding to the message topic, and it is determined that the permission verification fails, and the failure information of the publication can be fed back.

[0110] In one embodiment, as Figure 8 shown, step S301, that is, execute the target operation process corresponding to the target operation type to process the message processing request, including:

[0111] S801: Obtain the message push request, where the message push request includes the message topic and the push frequency;

[0112] S802: Send the message push request to the message consumer and wait for the consumption confirmation information returned by the candidate receiving message consumer within the target waiting time;

[0113] S803: If the message confirmation information is received within the target waiting time and the message confirmation information is a positive consumption information, push the message to be processed corresponding to the message topic to the message consumer according to the message frequency;

[0114] S804: If the message confirmation information is received within the target waiting time and the message confirmation information is a negative consumption information, then associate and record the message topic and the message consumer;

[0115] S805: If the message confirmation information is not received within the target waiting time, execute the resend push process and resend the message push request to the message consumer.

[0116] Among them, the message push request is a request for implementing message push. The message topic is the topic of the message to be pushed. The push frequency refers to the frequency of pushing messages at each interval. The message consumer is the user who consumes messages. The target waiting time is the pre-set waiting time. The consumption confirmation information is the confirmation information of the message consumer.

[0117] As an example, in step S801, the current Broker can obtain the message push request triggered by the user, and the message push topic shows the message topic to be pushed and the push frequency.

[0118] As an example, in step S802, after obtaining the message push request, the current Broker needs to send the message push request to the message consumer, so that the message consumer can determine whether to receive the messages related to the message topic pushed by the system at this push frequency according to the message topic and push frequency in the message push request. In this example, after sending the message push request to the message consumer, the current Broker needs to wait within the target waiting time to receive the consumption confirmation information returned by the message consumer, so as to determine whether the message consumer receives the message corresponding to the pushed message topic.

[0119] Among them, the positive consumption information is the information fed back by the message consumer agreeing to receive the push. The message to be processed refers to the message to be pushed to the message consumer.

[0120] As an example, in step S803, when the current Broker receives the message confirmation information within the target waiting time and the message confirmation information is positive consumption information, it indicates that the message consumer feeds back agreeing to receive the push. At this time, according to the message frequency, the message to be processed corresponding to the message topic can be pushed to the message consumer, so as to realize the targeted push of the message to be processed corresponding to a specific message topic to the message consumer related to this message topic, which helps to improve the pertinence of the message to be processed.

[0121] Among them, the negative consumption information refers to the information fed back by the message consumer disagreeing to receive the push.

[0122] As an example, in step S804, when the current Broker receives the message confirmation information within the target waiting time and the message confirmation information is negative consumption information, it indicates that the message consumer feeds back disagreeing to receive the push, and it is determined that the message consumer does not pay attention to the message corresponding to this message topic. At this time, the message topic and the message consumer are associated for subsequent statistical analysis of the message consumer and the message topics that the consumer does not pay attention to.

[0123] As an example, in step S805, when the current Broker does not receive the message confirmation information within the target waiting time, there may be a message delay. At this time, the retransmission push process is executed to resend the message push request to the message consumer. In this example, when the current Broker executes the retransmission push process, it is necessary to update the message retransmission times corresponding to each message consumer. When the message retransmission times do not reach the retransmission times threshold, the message push request is resent to the message consumer; when the message retransmission times reach the retransmission times threshold, it is assumed that the message consumer does not pay attention to the message corresponding to the message topic. At this time, the message topic and the message consumer are associated for subsequent statistical analysis of the message consumer and the message topics it does not pay attention to.

[0124] In one embodiment, as Figure 8 shown, step S301, that is, executing the target operation process corresponding to the target operation type to process the message processing request, includes:

[0125] (1) The current Broker pushes the message to be processed to the message consumer and waits to receive the confirmation consumption information or cancellation consumption information feedback by the message consumer within the preset waiting time.

[0126] (2) If the confirmation consumption information can be received within the preset waiting time, the message to be processed is deleted or retained. In this example, when the message consumer successfully consumes a message to be processed, the message consumer will send a confirmation consumption information to the current Broker.

[0127] (3) If the confirmation consumption information or cancellation consumption information is not received within the preset waiting time, or the cancellation consumption information is received within the preset waiting time, the message to be processed is resent to the consumer. In this example, the message to be processed is not successfully consumed. The current Broker automatically redelivers this message to be processed. The unacknowledged message automatic redelivery mechanism tracks all unacknowledged messages to be processed within the preset waiting time, that is, the message to be processed for which the confirmation consumption information is not received within the preset waiting time will be resent to the message consumer. When the message consumer fails to consume the message to be processed within the preset waiting time and wants to consume this message to be processed again, the message consumer can send a cancellation consumption information to the current Broker, and the Broker will resent this message to be processed to the message consumer.

[0128] In one embodiment, the cloud-native based message processing method further includes: (1) The current Broker receives a message to be processed sent by a message producer, where the message to be processed includes a message topic (Topic) and a sending mode. (2) The current Broker partitions and routes the message to be processed to a target Broker corresponding to the message topic (Topic) according to the sending mode, so as to update the target Broker as the new current Broker. (3) The target Broker (i.e., the new current Broker) determines its corresponding target subscription mode according to the message topic (Topic). (4) The target Broker (i.e., the new current Broker) sends the target message to the corresponding message consumer according to the target subscription mode. Wherein, the subscription mode is any one of exclusive, shared, disaster recovery, and key sharing.

[0129] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0130] In one embodiment, a cloud-native based message processing device is provided, and the cloud-native based message processing device corresponds one-to-one with the cloud-native based message processing method in the above embodiment. As Figure 9 shown, the cloud-native based message processing device includes a processing request receiving module 901, a message status monitoring module 902, a first processing module 903, and a second processing module 904. The detailed description of each functional module is as follows:

[0131] The processing request receiving module 901 is used to receive a message processing request, and the message processing request includes a target operation type;

[0132] The message status monitoring module 902 is used to monitor the current message status of the current message flow abstraction;

[0133] The first processing module 903 is used to execute a target operation process corresponding to the target operation type to process the message processing request when the current message status is a fault-free state;

[0134] The second processing module 904 is used to execute a message recovery process when the current message status is a faulty state, and execute a target operation process corresponding to the target operation type to process the message processing request when it is monitored that the current message status is a fault-free state.

[0135] In one embodiment, the cloud-native based message processing device includes:

[0136] The first write operation unit is used to obtain the message to be processed and the message processing type from the message processing request when the target operation type is the write operation type;

[0137] The status queue judgment unit is used to obtain the current batch identifier and the message sequence identifier corresponding to the message to be processed when the message processing type is the batch processing type, and judge whether there is a batch index status queue corresponding to the current batch identifier;

[0138] The first batch processing unit is used to write the message to be processed into the current Bookie if there is a batch index status queue, and update the current index status corresponding to the message sequence identifier in the batch index status queue;

[0139] The first batch processing unit is used to write the message to be processed into the current Bookie if there is no batch index status queue, create a batch index status queue corresponding to the current batch identifier, and record the current index status corresponding to the message sequence identifier;

[0140] The status queue release unit is used to monitor the current index status of the batch index status queue, and release the storage space corresponding to the batch index status queue when the current index status is the confirmed completion status.

[0141] In one embodiment, the cloud-native-based message processing device includes:

[0142] The second write operation unit is used to obtain the message to be processed and the message processing type from the message processing request when the target operation type is the write operation type;

[0143] The sharded message acquisition unit is used to obtain multiple sharded messages corresponding to the message to be processed when the message processing type is the sharded processing type;

[0144] The sharded message cache unit is used to store multiple sharded messages in the current cache;

[0145] The sharded message write unit is used to write all the sharded messages in the current cache into the current Bookie, and update the message transmission status of each sharded message in the current cache;

[0146] The current cache release unit is used to release the current cache if all the message transmission statuses are the sharded completion status.

[0147] In one embodiment, the cloud-native-based message processing device includes:

[0148] The storage duration monitoring unit is used to monitor the message storage duration corresponding to the message to be processed in the current Bookie;

[0149] A message migration processing unit, configured to execute a data migration process to migrate the to-be-processed messages in the current Bookie to disk space if the message storage duration is greater than the target duration threshold.

[0150] In one embodiment, a cloud-native-based message processing device includes:

[0151] A concurrent request acquisition unit, configured to receive a concurrent storage request formed based on to-be-processed messages;

[0152] A concurrent storage processing unit, configured to call a concurrent storage process according to the concurrent storage request and concurrently write the to-be-processed messages into the associated Bookies corresponding to a first quantity;

[0153] A feedback message statistics unit, configured to listen to the storage feedback messages returned by the associated Bookies and count a second quantity of the storage feedback messages;

[0154] A feedback ratio acquisition unit, configured to acquire a message feedback ratio according to the first quantity and the second quantity;

[0155] A concurrent process stop unit, configured to control the concurrent storage process to stop working if the message feedback ratio is greater than the target ratio threshold.

[0156] In one embodiment, a cloud-native-based message processing device includes:

[0157] A sending operation processing unit, configured to obtain the to-be-processed messages, the message topic, and the current producer from the message processing request when the target operation type is a publishing operation type;

[0158] A producer mode determination unit, configured to determine the producer mode corresponding to the message topic based on the message topic;

[0159] A first message publishing unit, configured to directly publish the to-be-processed messages if the producer mode is not the producer exclusive mode;

[0160] A second message sending unit, configured to obtain the target permission corresponding to the message topic and the current permission of the current producer if the producer mode is the producer exclusive mode, perform permission verification according to the target permission and the current permission, and publish the to-be-processed messages after the permission verification passes.

[0161] In one embodiment, a cloud-native-based message processing device includes:

[0162] A push request acquisition unit, which acquires a message push request, and the message push request includes a message topic and a push frequency;

[0163] A push request sending unit, configured to send a message push request to a message consumer, and wait for a consumption confirmation message returned by a candidate receiving message consumer within a target waiting time;

[0164] A first push processing unit, configured to, if a message confirmation message is received within the target waiting time and the message confirmation message is an affirmative consumption message, push a to-be-processed message corresponding to the message topic to the message consumer according to the message frequency;

[0165] A second push processing unit, configured to, if a message confirmation message is received within the target waiting time and the message confirmation message is a negative consumption message, associate and record the message topic and the message consumer;

[0166] A third push processing unit, configured to, if a message confirmation message is not received within the target waiting time, execute a retransmission push process and re-send the message push request to the message consumer.

[0167] For the specific limitations of the cloud-native-based message processing device, reference can be made to the limitations of the cloud-native-based message processing method in the above text, which will not be elaborated here. Each module in the above cloud-native-based message processing device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0168] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as Figure 10 shown. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data used or generated during the execution of the cloud-native-based message processing method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a cloud-native-based message processing method.

[0169] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the cloud-native-based message processing method in the above embodiment, such as Figure 2 shown in S201 - S204, or Figures 3 to 8As shown, to avoid repetition, it will not be elaborated here. Alternatively, when the processor executes the computer program, it implements the functions of the various modules / units in this embodiment of the cloud-native based message processing device. For example, the functions of the processing request receiving module 901, the message status monitoring module 902, the first processing module 903, and the second processing module 904 shown in FIG. 9. To avoid repetition, it will not be elaborated here.

[0170] In one embodiment, a computer-readable storage medium is provided. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements the cloud-native based message processing method in the above embodiment. For example Figure 2 S201 - S204 shown, or Figures 3 to 8 As shown, to avoid repetition, it will not be elaborated here. Alternatively, when the computer program is executed by a processor, it implements the functions of the various modules / units in this embodiment of the above cloud-native based message processing device. For example Figure 9 The functions of the processing request receiving module 901, the message status monitoring module 902, the first processing module 903, and the second processing module 904 shown. To avoid repetition, it will not be elaborated here.

[0171] Those of ordinary skill in the art can understand that all or part of the processes in implementing the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above various methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in this application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0172] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0173] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A cloud-native based message processing method, characterized in that, Including: Receiving a message processing request, where the message processing request includes a target operation type; Listening to the current message state of the current message stream abstraction; When the current message state is a fault-free state, executing a target operation process corresponding to the target operation type to process the message processing request; When the current message state is a faulty state, executing a message recovery process, and when it is monitored that the current message state is a fault-free state, executing a target operation process corresponding to the target operation type to process the message processing request; The executing the target operation process corresponding to the target operation type to process the message processing request includes: When the target operation type is a publish operation type, obtaining a message to be processed, a message topic, and a current producer from the message processing request; Based on the message topic, determining a producer mode corresponding to the message topic; If the producer mode is a producer exclusive mode, determining whether there is an existing producer for the message topic. If there is no existing producer for the message topic, associating the current producer with the message topic and publishing the message to be processed; if there is an existing producer for the message topic, obtaining a target permission corresponding to the message topic and a current permission of the current producer, performing permission verification based on the target permission and the current permission, and publishing the message to be processed after the permission verification passes; the producer exclusive mode means that only one producer exclusively uses a message topic, and the existing producer refers to a producer who sent other messages with the same message topic before the current moment.

2. The cloud-native based message processing method according to claim 1, wherein, The executing the target operation process corresponding to the target operation type to process the message processing request includes: When the target operation type is a write operation type, obtaining a message to be processed and a message processing type from the message processing request; When the message processing type is a batch processing type, obtaining a current batch identifier and a message sequence identifier corresponding to the message to be processed, and determining whether there is a batch index status queue corresponding to the current batch identifier; If there is the batch index status queue, writing the message to be processed into the current Bookie and updating the current index status corresponding to the message sequence identifier in the batch index status queue; If there is no such batch index status queue, writing the message to be processed into the current Bookie, creating a batch index status queue corresponding to the current batch identifier, and recording the current index status corresponding to the message sequence identifier; Listening to the current index status of the batch index status queue, and releasing the storage space corresponding to the batch index status queue when the current index status is a confirmed completion state.

3. The cloud-native based message processing method according to claim 1, wherein The executing the target operation process corresponding to the target operation type to process the message processing request includes: When the target operation type is a write operation type, obtaining a message to be processed and a message processing type from the message processing request; When the message processing type is the sharding processing type, obtain multiple sharded messages corresponding to the message to be processed; Store the multiple sharded messages in the current cache; Write all the sharded messages in the current cache into the current Bookie, and update the message transmission status of each sharded message in the current cache; If all the message transmission statuses are the sharding completion status, release the current cache.

4. The cloud-native based message processing method according to claim 1, wherein, The cloud-native based message processing method further includes: Monitor the message storage duration corresponding to the message to be processed in the current Bookie; If the message storage duration is greater than the target duration threshold, execute a data migration process to migrate the message to be processed in the current Bookie to disk space.

5. The cloud-native based message processing method according to claim 1, wherein The cloud-native based message processing method further includes: Receive a concurrent storage request formed based on the message to be processed; According to the concurrent storage request, call a concurrent storage process to concurrently write the message to be processed into the associated Bookies corresponding to a first quantity; Monitor the storage feedback messages returned by the associated Bookies, and count the second quantity of the storage feedback messages; Obtain a message feedback ratio according to the first quantity and the second quantity; If the message feedback ratio is greater than the target ratio threshold, control the concurrent storage process to stop working.

6. The cloud-native based message processing method according to claim 1, wherein The execution of the target operation process corresponding to the target operation type to process the message processing request includes: If the producer mode is not the producer exclusive mode, directly publish the message to be processed.

7. The cloud-native-based message processing method according to claim 1, wherein The execution of the target operation process corresponding to the target operation type to process the message processing request includes: Obtain a message push request, where the message push request includes a message topic and a push frequency; Send the message push request to a message consumer, and wait to receive a consumption confirmation message returned by the message consumer within a target waiting time; If a message confirmation message is received within the target waiting time, and the message confirmation message is an affirmative consumption message, push the message to be processed corresponding to the message topic to the message consumer according to the push frequency; If a message confirmation message is received within the target waiting time, and the message confirmation message is a negative consumption message, then associate and record the message topic and the message consumer; If a message confirmation message is not received within the target waiting time, execute a retransmission push process to resend the message push request to the message consumer.

8. A cloud-native based message processing device, characterized in that, Includes: A processing request receiving module, configured to receive a message processing request, where the message processing request includes a target operation type; A message status monitoring module, configured to monitor the current message status of the current message flow abstraction; A first processing module, configured to, when the current message status is a fault-free status, execute a target operation process corresponding to the target operation type to process the message processing request; A second processing module, configured to execute a message recovery process when the current message status is a faulty status, and execute a target operation process corresponding to the target operation type to process the message processing request when it is monitored that the current message status is a fault-free status; The cloud-native based message processing apparatus further includes: When the target operation type is a publishing operation type, obtain a message to be processed, a message topic, and a current producer from the message processing request; determine a producer mode corresponding to the message topic based on the message topic; if the producer mode is a producer exclusive mode, determine whether there is an existing producer for the message topic, if there is no existing producer for the message topic, associate the current producer with the message topic, and publish the message to be processed; if there is an existing producer for the message topic, obtain a target permission corresponding to the message topic and a current permission of the current producer, perform permission verification according to the target permission and the current permission, and publish the message to be processed after the permission verification passes; the producer exclusive mode refers to a mode in which only one producer exclusively uses a message topic, and the existing producer refers to a producer who has sent other messages with the same message topic before the current moment.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the cloud-native based message processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the cloud-native based message processing method according to any one of claims 1 to 7.

Images