Data processing method and device based on Redis Stream and computer equipment-level medium

Abstract

The invention relates to a data processing method and device based on Redis Stream and a computer equipment-level medium, and the method comprises the steps: defining a namespace, a subject name and a configuration item, creating a subject information key in Redis according to the namespace, the subject name and the configuration item, receiving data written into a subject by a producer, extracting a partition key transmitted by the producer, and storing the data in the partition key; according to the theme configuration information and the partition key, a partition ID corresponding to the write-in data is calculated, and the write-in data is sent to a Stream of the corresponding partition ID; defining a consumption group name, and creating a consumer group information key in the Redis according to the namespace, the theme name and the consumption group name; responding to a consumer change event, and generating a new partition distribution scheme by adopting a partition rebalance mechanism according to the consumer group information key; and on the basis of the new partition distribution scheme, updating partition locks corresponding to different consumers, and controlling the consumers to consume Stream data of corresponding partitions according to the partition locks corresponding to the consumers.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, storage medium and computer program product based on Redis Stream. Background Art

[0002] In distributed systems, message queues, as an important middleware, are widely used in scenarios such as asynchronous data processing, system decoupling, and traffic peak shaving.

[0003] Redis Streams is an advanced data structure provided by Redis that supports message persistence, publish / subscribe, and consumer groups, providing developers with a flexible message processing mechanism.

[0004] However, Redis Streams has limitations in terms of the order of message processing. Specifically, although Redis Streams provides a consumer group function that allows multiple consumers to collaboratively process the same message stream, all messages in traditional Redis Streams are stored in the same logical stream. In this case, when multiple consumers consume messages at the same time, there is no guarantee that the messages will be processed in the order in which they were sent. Summary of the invention

[0005] Based on this, it is necessary to provide a data processing method, device, computer equipment, storage medium and computer program product based on RedisStream that can ensure orderly processing of messages in response to the above technical problems.

[0006] In a first aspect, the present application provides a data processing method based on Redis Stream. The method comprises:

[0007] Define a namespace, topic name, and configuration items, and create a topic information key in Redis according to the namespace, topic name, and configuration items. The topic information key is used to store topic configuration information.

[0008] Receive the data written by the producer to the topic, extract the partition key passed in by the producer, calculate the partition ID corresponding to the written data according to the topic configuration information and the partition key, and send the written data to the Stream corresponding to the partition ID;

[0009] Define a consumer group name, and create a consumer group information key in Redis according to the namespace, the topic name, and the consumer group name. The consumer group information key is used to store a consumer list and a partition allocation scheme.

[0010] In response to a consumer change event, a new partition allocation scheme is generated according to the consumer group information key and using a partition rebalancing mechanism;

[0011] Based on the new partition allocation scheme, update the partition locks corresponding to different consumers, and control consumers to consume the Stream data of the corresponding partitions according to the partition locks corresponding to the consumers.

[0012] In one embodiment, responding to the consumer change event, generating a new partition allocation scheme according to the consumer group information key and using a partition rebalancing mechanism includes:

[0013] When a consumer joins, check whether the topic corresponding to the consumer to be added exists;

[0014] If it exists, obtain the consumer group management lock corresponding to the consumer to be added;

[0015] Generate a new partition allocation scheme according to the consumer group information key and using a partition rebalancing mechanism, and update the consumer group information key according to the new partition allocation scheme;

[0016] Start the associated subtask corresponding to the consumer to be added, and send control information of the newly added consumer;

[0017] Release the consumer group management lock.

[0018] In one embodiment, responding to the consumer change event, generating a new partition allocation scheme according to the consumer group information key and using a partition rebalancing mechanism includes:

[0019] When a consumer leaves, check whether the topic corresponding to the departing consumer exists;

[0020] If it exists, obtain the consumer group management lock corresponding to the leaving consumer;

[0021] Close the associated subtask corresponding to the leaving consumer;

[0022] Generate a new partition allocation scheme according to the consumer group information key and using a partition rebalancing mechanism, and update the consumer group information key according to the new partition allocation scheme;

[0023] Send control information to remove the leaving consumer and release the consumer group management lock.

[0024] In one of the embodiments, the associated subtasks include heartbeat lock, periodic check for crashed consumers, and subscription control flow.

[0025] In one embodiment, responding to the consumer change event, generating a new partition allocation scheme according to the consumer group information key and using a partition rebalancing mechanism includes:

[0026] In response to a consumer change event, identifying a first informed person corresponding to the consumer change event;

[0027] The first insider obtains a list of currently active consumers from the consumer group information key;

[0028] Re-allocating the mapping relationship between consumers and partitions according to the currently active consumer list and a preset allocation strategy, wherein the preset allocation strategy is generated based on load balancing and / or reducing excessive switching of consumers to partitions;

[0029] The first informed party updates the mapping relationship between the reallocated consumers and partitions into the consumer group information key.

[0030] In one embodiment, the reallocation of the mapping relationship between consumers and partitions according to the currently active consumer list and the preset allocation strategy includes:

[0031] According to the currently active consumer list, obtain the number of currently active consumers N and the number of partitions M;

[0032] According to the current number of active consumers N and the number of partitions M, determine the maximum number of partitions to be evenly distributed to each consumer;

[0033] Consumers are sorted according to the time they joined the consumer group. Based on the sorting results and the maximum number of partitions allocated to each consumer, the mapping relationship between consumers and partitions is reallocated using a continuous partition allocation method.

[0034] In one embodiment, responding to a consumer change event and identifying a first insider corresponding to the consumer change event includes:

[0035] Respond to consumer change events and identify the corresponding types of consumer change events;

[0036] If the corresponding type of the consumer change event is consumer joining, the newly joined consumer is determined to be the first informed person;

[0037] If the corresponding type of the consumer change event is consumer leaving, the departing consumer is determined to be the first informed person;

[0038] If the corresponding type of the consumer change event is consumer crash, the first consumer who scans and finds the crashed consumer is determined to be the first informed party.

[0039] In a second aspect, the present application also provides a data processing device based on Redis Stream. The device comprises:

[0040] A topic creation module is used to define a namespace, a topic name, and configuration items, and to create a topic information key in Redis according to the namespace, the topic name, and configuration items. The topic information key is used to store topic configuration information.

[0041] The data writing module is used to receive the data written by the producer to the topic, extract the partition key passed in by the producer, calculate the partition ID corresponding to the written data according to the topic configuration information and the partition key, and send the written data to the Stream corresponding to the partition ID;

[0042] A consumer group creation module, used to define a consumer group name, and create a consumer group information key in Redis according to the namespace, the topic name, and the consumer group name, wherein the consumer group information key is used to store a consumer list and a partition allocation scheme;

[0043] A partition module is used to respond to consumer change events and generate a new partition allocation plan based on the consumer group information key and using a partition rebalancing mechanism;

[0044] The control module is used to update the partition locks corresponding to different consumers based on the new partition allocation scheme, and control consumers to consume the Stream data of the corresponding partitions according to the partition locks corresponding to the consumers.

[0045] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0046] Define a namespace, topic name, and configuration items, and create a topic information key in Redis according to the namespace, topic name, and configuration items. The topic information key is used to store topic configuration information.

[0047] Receive the data written by the producer to the topic, extract the partition key passed in by the producer, calculate the partition ID corresponding to the written data according to the topic configuration information and the partition key, and send the written data to the Stream corresponding to the partition ID;

[0048] Define a consumer group name, and create a consumer group information key in Redis according to the namespace, the topic name, and the consumer group name. The consumer group information key is used to store a consumer list and a partition allocation scheme.

[0049] In response to a consumer change event, a new partition allocation scheme is generated according to the consumer group information key and using a partition rebalancing mechanism;

[0050] Based on the new partition allocation scheme, update the partition locks corresponding to different consumers, and control consumers to consume the Stream data of the corresponding partitions according to the partition locks corresponding to the consumers.

[0051] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0052] Define a namespace, topic name, and configuration items, and create a topic information key in Redis according to the namespace, topic name, and configuration items. The topic information key is used to store topic configuration information.

[0053] Receive the data written by the producer to the topic, extract the partition key passed in by the producer, calculate the partition ID corresponding to the written data according to the topic configuration information and the partition key, and send the written data to the Stream corresponding to the partition ID;

[0054] Define a consumer group name, and create a consumer group information key in Redis according to the namespace, the topic name, and the consumer group name. The consumer group information key is used to store a consumer list and a partition allocation scheme.

[0055] In response to a consumer change event, a new partition allocation scheme is generated according to the consumer group information key and using a partition rebalancing mechanism;

[0056] Based on the new partition allocation scheme, update the partition locks corresponding to different consumers, and control consumers to consume the Stream data of the corresponding partitions according to the partition locks corresponding to the consumers.

[0057] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0058] Define a namespace, topic name, and configuration items, and create a topic information key in Redis according to the namespace, topic name, and configuration items. The topic information key is used to store topic configuration information.

[0059] Receive the data written by the producer to the topic, extract the partition key passed in by the producer, calculate the partition ID corresponding to the written data according to the topic configuration information and the partition key, and send the written data to the Stream corresponding to the partition ID;

[0060] Define a consumer group name, and create a consumer group information key in Redis according to the namespace, the topic name, and the consumer group name. The consumer group information key is used to store a consumer list and a partition allocation scheme.

[0061] In response to a consumer change event, a new partition allocation scheme is generated according to the consumer group information key and using a partition rebalancing mechanism;

[0062] Based on the new partition allocation scheme, update the partition locks corresponding to different consumers, and control consumers to consume the Stream data of the corresponding partitions according to the partition locks corresponding to the consumers.

[0063] The above-mentioned data processing method, device, computer equipment, storage medium and computer program product based on Redis Stream define namespace, topic name and configuration items, create topic information key in Redis according to namespace, topic name and configuration items, and the topic information key is used to store topic configuration information; receive data written by producers to the topic, extract partition key passed by producers, calculate partition ID corresponding to written data according to topic configuration information and partition key, and send written data to Stream corresponding to partition ID; define consumer group name, create consumer group information key in Redis according to namespace, topic name and consumer group name, and consumer group information key is used to store consumer list and partition allocation scheme; respond to consumer change event, generate new partition allocation scheme according to consumer group information key and partition rebalancing mechanism; based on the new partition allocation scheme, update partition locks corresponding to different consumers, and control consumers to consume Stream data of corresponding partitions according to partition locks corresponding to consumers. In the whole process, for Redis Stream data, Redis data is orderly entered into topic partitions, which ensures orderly processing and rebalancing mechanism of partition data by consumers, and no additional heavyweight components are introduced, so that orderly processing of Redis Stream data is realized at low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is an application environment diagram of a data processing method based on Redis Stream in an embodiment;

[0065] Figure 2 The figure is a flowchart of a data processing method based on Redis Stream in one embodiment;

[0066] Figure 3 is a schematic diagram of a sub-process of S300 in an embodiment;

[0067] Figure 4 A flow chart for periodically checking for crashed consumers;

[0068] Figure 5 It is a flowchart of control flow processing;

[0069] Figure 6A schematic diagram of the processing flow for consumer joining in a specific application example;

[0070] Figure 7 is a schematic diagram of a sub-process of S300 in another embodiment;

[0071] Figure 8 It is a schematic diagram of the processing flow of a consumer leaving in a specific application example;

[0072] Fig. 9 This is a schematic diagram of the overall storage structure of Redis;

[0073] Fig.10 is a structural block diagram of a data processing device based on Redis Stream in an embodiment;

[0074] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0076] In order to explain in detail the technical principles and technical effects of the data processing method based on Redis Stream in this application, the following will first introduce the relevant computer underlying data processing related contents.

[0077] Redis Streams: Stream is a new data structure added in Redis 5.0. It is a high-performance, persistent message queue that can be used to publish and subscribe to messages.

[0078] Topic: It is a logical concept in message queue systems (such as Kafka) used to classify and organize messages. It is similar to a table in a database or a folder in a file system. All messages related to a topic are published to the topic.

[0079] Partition: A mechanism in a message queue system (such as Kafka) that is used to split the message data in a topic into multiple independent subsets that can be processed in parallel. Each partition is an ordered message queue with an independent log file.

[0080] Consumer Group: In a message queue system (such as Kafka and Redis Streams), a group of consumers work together to consume data in a message queue in a load-balanced and fault-tolerant manner.

[0081] Redis Distributed Lock: A distributed lock mechanism based on Redis, used to coordinate access to shared resources by multiple processes in a distributed system. Distributed locks can prevent multiple processes from modifying shared resources at the same time, thereby avoiding data inconsistencies and race conditions.

[0082] Rebalance: refers to the process in which Kafka reallocates partitions to consumers when the number of consumers or the number of partitions changes in a message queue system (such as Kafka) consumer group.

[0083] Kafka is a distributed event stream processing platform developed and open-sourced by LinkedIn. It is designed for processing large-scale real-time data streams and has high throughput, persistent storage, horizontal expansion, high availability and real-time stream processing capabilities. Redis added a stream data structure in version 5.0. Redis Stream is conceptually equivalent to a single partition of a Kafka topic. The two have the following similarities: 1) Both provide persistent and ordered event storage; 2) Event storage uses key-value format; 3) Support publish-subscribe mode, where producers publish messages and multiple consumers subscribe and process messages; 4) Allow multiple consumers to subscribe to the same topic / stream at the same time to achieve parallel processing. However, compared with Kafka, Redis Streams is different in design concept and implementation. Redis is an in-memory database. Although Stream supports persistence, the main data is stored in memory, which is suitable for short-term and low-latency data processing. Kafka is designed for large-scale data stream processing. Data is mainly stored on disk, and efficient persistent storage is achieved through sequential write and zero-copy technology, which is suitable for long-term data storage and high throughput requirements. Redis's messaging and consumption model is relatively simple, supporting basic publish-subscribe mode and consumer groups. Kafka provides rich message delivery semantics, including exactly once delivery, guaranteed order delivery, message backtracking, and transaction support, which is suitable for complex data stream processing requirements.

[0084] Further research found that although Redis Streams provides consumer group functions, it cannot guarantee the orderly processing of messages in the case of multiple consumers due to the lack of data partitioning. This application proposes a message queue implementation with Kafka-like semantics based on the ecological characteristics of Redis. The entire solution aims to implement stream data partitioning on Redis Streams, and orderly processing and rebalancing of data by consumers in the same consumer group.

[0085] Based on the above technical principles, the data processing method based on Redis Stream provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. Terminal 102 sends a data processing request of Redis Stream to server 104, and server 104 responds to the data processing request. Server 104 defines a namespace, a topic name, and configuration items, and creates a topic information key in Redis according to the namespace, the topic name, and the configuration items. The topic information key is used to store the topic configuration information; receives the data written by the producer to the topic, extracts the partition key passed in by the producer, calculates the partition ID corresponding to the written data according to the topic configuration information and the partition key, and sends the written data to the Stream of the corresponding partition ID; defines the consumer group name, creates a consumer group information key in Redis according to the namespace, the topic name, and the consumer group name, and the consumer group information key is used to store the consumer list and the partition allocation plan; responds to consumer change events, generates a new partition allocation plan according to the consumer group information key and adopts the partition rebalancing mechanism; based on the new partition allocation plan, updates the partition locks corresponding to different consumers, and controls the consumer to consume the Stream data of the corresponding partition according to the partition locks corresponding to the consumer. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0086] In one embodiment, Figure 2 As shown in the figure, a data processing method based on Redis Stream is provided. Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0087] S100: define a namespace, a topic name, and configuration items, and create a topic information key in Redis according to the namespace, the topic name, and the configuration items. The topic information key is used to store topic configuration information.

[0088] Namespace: used to distinguish data in different applications or business scenarios to prevent data conflicts. Topic name: represents the logical classification of data, such as user behavior data, transaction data, etc. Configuration items: including the number of partitions, partition strategy (such as hash partition, range partition, etc.), data retention strategy, etc. Based on this information, create a topic information key (such as namespace: topic: config) in Redis to store the above configuration information.

[0089] Specifically, a subject object is defined here: the method signature is as follows:

[0090] Topic newTopic(String namespace,String topicName,Map<String,Object> config)

[0091] Among them, namespace is the namespace; topicName is the topic name; config is the configuration item; the required configuration is the number of topic partitions partitionNum; the optional partition length partitionSize; the partition hash algorithm hashAlg; if not filled, md5 is used by default. Each topic creates a key (topic information key) on redis, and the key name consists of the namespace and the topic name. The value stores the creation configuration information of the topic.

[0092] topic=_namespace:topicName:info

[0093] Where namespace is the namespace; topicName is the topic name. Each partition of the topic corresponds to a RedisStreams, and the stream is named _namespace: topicName: partition_{id}, where {id} is the partition value (0-partitionNum-1). The stream does not need to be created in advance.

[0094] S200: Receive the data written by the producer to the topic, extract the partition key passed in by the producer, calculate the partition ID corresponding to the written data according to the topic configuration information and the partition key, and send the written data to the Stream corresponding to the partition ID.

[0095] The producer data specifically contains the data fields and partition keys required by the business logic. Extracting the partition key refers to extracting the key fields used for partitioning from the producer data. Calculating the partition ID refers to using the corresponding partitioning algorithm to calculate the partition ID to which the data should be written based on the topic configuration information and the partition key. Writing to a stream refers to writing data to the corresponding partition of a Redis stream, using the XADD command and specifying the partition ID as part of the stream key. Specifically, when the producer writes data to a topic, it passes in the partition key. The method signature is: String send(Topic topic, StringpartitionKey, Tobject). According to the configuration of the topic, the number of partitions partitionNum is obtained, and based on the configured hashAlg, the partition ID corresponding to the corresponding data is calculated. partitionId = hashAlg(partitionKey) % partitionNum. After the data object is serialized, it is sent to the stream corresponding to the partition ID partitionId.

[0096] S300: Define the consumer group name, and create a consumer group information key in Redis according to the namespace, topic name, and consumer group name. The consumer group information key is used to store the consumer list and partition allocation plan.

[0097] The consumer group name is used to identify a group of consumers that work together. Creating a consumer group information key means creating a key in Redis (such as namespace:topic:consumer_group:info) to store the consumer list and the current partition allocation scheme. Initializing partition allocation means allocating partitions to consumers based on the number of consumers and the number of partitions using a certain strategy (such as polling, hashing, etc.). Specifically, a topic consumer group corresponds to one topic. Redis stipulates that each consumer group must only be unique in the same Stream. Therefore, a topic consumer group can establish a Stream consumer group with the same name on the Stream corresponding to all the partitions below. This embodiment takes consumerGroup as an example. Each topic consumer group maintains a consumer group-level key (consumer group information key) on Redis, and the key name consists of the namespace, topic name, and consumer group name. The value stores the consumer list and partition allocation scheme of the current consumer group.

[0098] key=__namespace:topicName:consumerGroup:info.

[0099] Where namespace is the namespace, topicName is the topic name, and consumerGroup is the consumer name. An example of the allocation plan is as follows:

[0100] [{"consumerId":"consumer1","joinedAt":1800000000000,"partitions":[0,1]},{"consumerId":"consumer2","joinedAt":1800000000000,"partitions":[2]}].

[0101] Where consumerId is the unique identifier of the consumer, partitions is the assigned partition id, and joinedAt is the unix time when the consumer joins the consumer group. The allocation scheme algorithm mentioned later can be used.

[0102] S400: In response to a consumer change event, a new partition allocation plan is generated according to the consumer group information key and using a partition rebalancing mechanism.

[0103] Consumer change events include consumer joining, consumer exiting, or consumer crashing. The partition rebalancing mechanism is triggered when a consumer changes, and the partitions that each consumer is responsible for are recalculated. Collect the current state: Get the current consumer list and partition allocation plan. Calculate the new allocation plan. Specifically, a more balanced allocation strategy can be used to generate a new partition allocation plan based on the number of consumers and the number of partitions. Updating consumer group information specifically means writing the new partition allocation plan into the consumer group information key.

[0104] S500: Based on the new partition allocation scheme, the partition locks corresponding to different consumers are updated, and according to the partition locks corresponding to the consumers, the consumers are controlled to consume the Stream data of the corresponding partitions.

[0105] Partition locks are used to ensure that each partition can only be processed by one consumer at the same time to prevent duplicate data consumption. Update partition locks are used to update the partition locks corresponding to each consumer according to the new partition allocation plan. Controlled consumption means that consumers read and process data from the corresponding Redis Stream partitions according to the partition locks they are responsible for. Use the XREADGROUP command to read data, and use the XACK command to confirm the processed data. In other words, when consumers consume partition stream data, they first consume the data in the pending state in the stream, and then consume normal data from the stream.

[0106] The above-mentioned data processing method based on Redis Stream defines the namespace, topic name and configuration items, creates a topic information key in Redis according to the namespace, topic name and configuration items, and the topic information key is used to store the topic configuration information; receives the data written by the producer to the topic, extracts the partition key passed by the producer, calculates the partition ID corresponding to the written data according to the topic configuration information and the partition key, and sends the written data to the Stream of the corresponding partition ID; defines the consumer group name, creates a consumer group information key in Redis according to the namespace, topic name and consumer group name, and the consumer group information key is used to store the consumer list and partition allocation scheme; responds to consumer change events, generates a new partition allocation scheme according to the consumer group information key and adopts the partition rebalancing mechanism; based on the new partition allocation scheme, updates the partition locks corresponding to different consumers, and controls the consumer to consume the Stream data of the corresponding partition according to the partition locks corresponding to the consumer. In the whole process, for Redis Stream data, the orderly entry of Redis data into the topic partition is realized, ensuring the orderly processing and rebalancing mechanism of the partition data by consumers, and no additional heavyweight components are introduced, so that the orderly processing of Redis Stream data is realized at a low cost.

[0107] In one embodiment, if Figure 3 As shown, S300 includes:

[0108] S311: When a consumer joins, check whether the topic corresponding to the consumer to be added exists.

[0109] When the system receives a request to join a consumer, the first task is to check whether the topic specified in the request already exists in the system. This step is critical because it ensures that the consumer can join a valid, configured topic.

[0110] S312: If it exists, obtain the consumer group management lock corresponding to the consumer to be added.

[0111] Once the topic is confirmed to exist, the system needs to obtain the management lock of the consumer group. This lock mechanism is designed to ensure that there will be no interference from other concurrent operations when updating consumer group information, thereby maintaining data consistency and integrity.

[0112] S313: Generate a new partition allocation plan based on the consumer group information key and the partition rebalancing mechanism, and update the consumer group information key according to the new partition allocation plan.

[0113] Based on the current consumer group information (including the number of existing consumers and the partitions each consumer is currently responsible for) and the newly added consumer information, the system uses the partition rebalancing mechanism to calculate a new partition allocation scheme. This scheme aims to achieve load balancing between consumers while minimizing data redistribution. After calculating the new partition allocation scheme, the system will update the corresponding consumer group information key in Redis to reflect the latest partition allocation.

[0114] S314: Start the associated subtask corresponding to the consumer to be added, and send control information of the newly added consumer.

[0115] For newly added consumers, the server may need to start some associated subtasks or threads so that the consumer can start processing the data in the partition it is responsible for. Specifically, the subtasks mainly include heartbeat locks, timed checks for crashed consumers, and subscription control flows. Among them, heartbeat locks: hold consumer-level distributed locks on redis. The lock name consists of the namespace, topic name, consumer group name, and consumer id: csLock = namespace: topicName: consumerGroup: consumerId: lock. Among them, namespace is the namespace, topicName is the topic name, consumerGroup is the consumer name, and consumerId is the consumer ID.

[0116] Regularly check crashed consumers: This thread regularly checks the heartbeat lock consumers according to the consumer list in the consumer group information key. If it is found that a consumer does not exist, it regenerates the partition allocation plan and pushes the consumer offline message to the control flow. The specific process is as follows: Figure 4 Subscription control flow: Get the partition scheme change message from the control flow of the consumer group, read the consumer group information key to get the latest partition scheme. Check the partitions currently held by the consumer. If there are reduced partitions, release the partition lock; if there are new partitions, try to get the topic consumer group partition lock. Since the lock needs to wait for the previous consumer to release it, it is necessary to increase the retry. The process is as follows Figure 5 shown.

[0117] S315: Release the consumer group management lock.

[0118] After completing all the above steps, the system will release the consumer group management lock previously acquired. This step ensures that other operations that may be waiting for lock resources can continue, thereby improving the system's concurrent processing capabilities. Further, in one of the specific application examples, for the case of consumer joining, the overall processing flow is as follows Figure 6 shown.

[0119] In one embodiment, if Figure 7 As shown, S300 includes:

[0120] S321: When a consumer leaves, check whether the topic corresponding to the leaving consumer exists.

[0121] When a consumer is detected to have left (perhaps because the consumer actively disconnected, crashed, or was removed by an administrator), the first task is to confirm whether the topic to which the consumer belongs still exists in the system. This step ensures the validity of subsequent operations.

[0122] S322: If it exists, obtain the consumer group management lock corresponding to the leaving consumer.

[0123] Once the topic is confirmed to exist, the server needs to obtain the management lock of the consumer group. This step is to ensure that there will be no interference from other concurrent operations when updating the consumer group information, thereby maintaining the consistency and integrity of the data.

[0124] S323: Close the associated subtask corresponding to the departing consumer.

[0125] After acquiring the management lock, the server needs to close all associated subtasks corresponding to the departing consumer. These subtasks may include data reading, processing, persistence, etc. Closing these subtasks can ensure the correct release of resources and prevent resource leakage or data inconsistency caused by the departure of consumers.

[0126] S324: Generate a new partition allocation plan based on the consumer group information key and the partition rebalancing mechanism, and update the consumer group information key based on the new partition allocation plan.

[0127] Based on the current consumer group information (including the number of remaining consumers and the partitions each consumer is currently responsible for) and the information of the departing consumers, the server uses the partition rebalancing mechanism to calculate a new partition allocation scheme. This step aims to reallocate the partitions that the departing consumers are responsible for in order to achieve load balancing between consumers. After calculating the new partition allocation scheme, the server updates the corresponding consumer group information key in Redis to reflect the latest partition allocation. This step is visible to other consumers and system components, so they can make corresponding adjustments based on the new allocation scheme.

[0128] S325: Send control information to remove the departing consumer and release the consumer group management lock.

[0129] The server needs to send control information to other consumers or system components to inform them that the departing consumer has been removed and the new partition allocation scheme has taken effect. This step helps ensure that all parts of the system remain synchronized and consistent. After completing all the above steps, the server releases the consumer group management lock previously acquired. This step ensures that other operations that may be waiting for lock resources can continue, thereby improving the concurrent processing capabilities of the system. Specifically, in a specific application embodiment, for the case where a consumer leaves, the overall processing flow is as follows: Figure 8 shown.

[0130] In one embodiment, in response to a consumer change event, generating a new partition allocation scheme according to a consumer group information key and using a partition rebalancing mechanism includes:

[0131] Step 1: Respond to a consumer change event and identify the first informed party corresponding to the consumer change event.

[0132] When the server detects a consumer change event (such as a consumer joining, leaving, or failure), it first needs to determine which component or process is the first informed party of the change. The purpose of determining the first informed party is to ensure that it can obtain and process information about consumer changes in a timely and accurate manner. Specifically, in response to a consumer change event, the corresponding type of the consumer change event is identified; if the corresponding type of the consumer change event is a consumer joining, the newly joined consumer is determined to be the first informed party; if the corresponding type of the consumer change event is a consumer leaving, the departing consumer is determined to be the first informed party; if the corresponding type of the consumer change event is a consumer crash, the first consumer who scans and discovers the crashed consumer is determined to be the first informed party.

[0133] Step 2: The first informant obtains the list of currently active consumers from the consumer group information key.

[0134] Once the first informed party is determined, it reads the currently active consumer list from the consumer group information key in Redis. This list contains all consumers currently connected to the topic and processing data. Obtaining this list is the basis for reassigning the mapping relationship between consumers and partitions.

[0135] Step 3: Reassign the mapping relationship between consumers and partitions according to the current active consumer list and the preset allocation strategy. The preset allocation strategy is generated based on load balancing and / or reducing excessive switching of consumers to partitions.

[0136] Next, the first informed party will reallocate the mapping relationship between consumers and partitions based on the current active consumer list and the preset allocation strategy. This preset allocation strategy may be based on load balancing (such as allocating more partitions to consumers with stronger processing capabilities), reducing excessive switching of consumers to partitions (such as avoiding frequently allocating the same partition to different consumers), or other business logic requirements. The purpose of reallocating mapping relationships is to ensure the efficiency and reliability of data processing while minimizing system fluctuations caused by consumer changes.

[0137] Step 4: The first informed party updates the mapping relationship between the reallocated consumers and partitions into the consumer group information key.

[0138] Finally, the first informed party updates the mapping of the reallocated consumers to partitions into the consumer group information key in Redis. This step is visible to other consumers and system components, so they can make corresponding adjustments based on the new mapping. After the consumer group information key is updated, the system can process data according to the new allocation scheme.

[0139] In one embodiment, reallocating the mapping relationship between consumers and partitions according to the currently active consumer list and the preset allocation strategy includes:

[0140] Step 1: According to the current active consumer list, obtain the current number of active consumers N and the number of partitions M.

[0141] First, the server will obtain the number of currently active consumers N and the number of partitions to be allocated M based on the currently active consumer list. These two parameters are the basis for reallocating the mapping relationship between consumers and partitions.

[0142] Step 2: Based on the current number of active consumers N and the number of partitions M, determine the maximum number of partitions that each consumer should be evenly allocated.

[0143] Next, based on the current number of active consumers N and the number of partitions M, determine the maximum number of partitions that each consumer can be allocated on average. This value can be obtained by dividing the number of partitions M by the number of consumers N (rounding may be required). It should be noted that this average allocation number is only an upper limit or reference value, and the actual allocation may need to be adjusted based on other factors (such as consumer processing power, network latency, etc.).

[0144] Step 3: Sort consumers by the time they joined the consumer group, and reallocate the mapping relationship between consumers and partitions based on the sorting results and the maximum number of partitions evenly allocated to each consumer and the continuous allocation of partitions.

[0145] Consumers are sorted according to the time when they joined the consumer group. This sorting step is to take into account the "first come, first served" principle of consumers when allocating partitions, so as to maintain the stability of the allocation to a certain extent. Of course, this sorting step is not necessary. Whether to use and what sorting method to use needs to be determined based on business needs and system design. Finally, based on the sorting results and the maximum number of partitions allocated to each consumer on average, the mapping relationship between consumers and partitions is reallocated by using the method of continuous partition allocation. This allocation method means that each consumer will be assigned a continuous range of partitions, thereby reducing the overhead caused by partition switching between consumers. Of course, in actual applications, this continuous allocation method may need to be adjusted and optimized according to specific business needs and system load conditions.

[0146] In the specific application example, the partition allocation scheme is generated according to the consumer group configuration. The default algorithm arranges the consumers in ascending order according to the time they joined the consumer group, and then evenly distributes them according to the number of partitions. Assuming there are N consumers and M partitions, the maximum number of partitions that each consumer can evenly distribute is P = MAX(1, INT(M / N)). In order to reduce partition adjustments, after consumers are sorted, partitions are continuously allocated, that is, the first consumer 1 allocates partitions 0 to P-1, and then allocates P to 2P-1 to the second consumer.

[0147] In a specific application example, the overall storage structure of Redis in this application is as follows: Fig. 9 shown.

[0148] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0149] Based on the same inventive concept, the embodiment of the present application also provides a data processing device based on Redis Stream for implementing the data processing method based on Redis Stream involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the data processing device based on Redis Stream provided below can refer to the limitations of the data processing method based on Redis Stream above, and will not be repeated here.

[0150] In one embodiment, Fig.10 As shown, a data processing device based on Redis Stream is provided, including:

[0151] The topic creation module 100 is used to define a namespace, a topic name and configuration items, and to create a topic information key in Redis according to the namespace, the topic name and configuration items. The topic information key is used to store topic configuration information.

[0152] The data writing module 200 is used to receive the data written by the producer to the topic, extract the partition key passed in by the producer, calculate the partition ID corresponding to the written data according to the topic configuration information and the partition key, and send the written data to the Stream corresponding to the partition ID;

[0153] The consumer group creation module 300 is used to define the consumer group name, and create a consumer group information key in Redis according to the namespace, topic name and consumer group name. The consumer group information key is used to store the consumer list and partition allocation scheme;

[0154] Partition module 400, used to respond to consumer change events, generate new partition allocation schemes according to consumer group information keys and adopt partition rebalancing mechanism;

[0155] The control module 500 is used to update the partition locks corresponding to different consumers based on the new partition allocation scheme, and control the consumers to consume the Stream data of the corresponding partitions according to the partition locks corresponding to the consumers.

[0156] In one of the embodiments, the partition module 400 is also used to detect whether the topic corresponding to the consumer to be added exists when a consumer joins; if it exists, obtain the consumer group management lock corresponding to the consumer to be added; generate a new partition allocation plan based on the consumer group information key and the partition rebalancing mechanism, and update the consumer group information key according to the new partition allocation plan; start the associated subtask corresponding to the consumer to be added, and send the control information of the newly added consumer; release the consumer group management lock.

[0157] In one of the embodiments, the partition module 400 is also used to detect whether the topic corresponding to the leaving consumer exists when a consumer leaves; if so, obtain the consumer group management lock corresponding to the leaving consumer; close the associated subtasks corresponding to the leaving consumer; generate a new partition allocation plan based on the consumer group information key and the partition rebalancing mechanism, and update the consumer group information key based on the new partition allocation plan; send control information to remove the leaving consumer and release the consumer group management lock.

[0158] In one embodiment, the associated subtasks include heartbeat locks, periodic checks for crashed consumers, and subscription control flows.

[0159] In one embodiment, the partition module 400 is also used to respond to a consumer change event and identify a first insider corresponding to the consumer change event; the first insider obtains a currently active consumer list from a consumer group information key; reallocates the mapping relationship between consumers and partitions based on the currently active consumer list and a preset allocation strategy, the preset allocation strategy is generated based on load balancing and / or reducing excessive switching of consumers to partitions; the first insider updates the mapping relationship between the reallocated consumers and partitions to the consumer group information key.

[0160] In one of the embodiments, the partitioning module 400 is also used to obtain the current number of active consumers N and the number of partitions M based on the current active consumer list; determine the maximum number of partitions to be evenly distributed to each consumer based on the current number of active consumers N and the number of partitions M; sort the consumers according to the time when they joined the consumer group, and reallocate the mapping relationship between consumers and partitions based on the sorting results and the maximum number of partitions to be evenly distributed to each consumer, using a method of continuous partition allocation.

[0161] In one embodiment, the partitioning module 400 is also used to respond to consumer change events and identify the corresponding type of consumer change events; if the corresponding type of the consumer change event is consumer joining, the newly joined consumer is determined to be the first informed party; if the corresponding type of the consumer change event is consumer leaving, the departing consumer is determined to be the first informed party; if the corresponding type of the consumer change event is consumer crash, the first consumer who scans and finds the crashed consumer is determined to be the first informed party.

[0162] Each module in the above-mentioned data processing device based on Redis Stream can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0163] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store preset data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method based on Redis Stream is implemented.

[0164] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0165] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned data processing method based on Redis Stream when executing the computer program.

[0166] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data processing method based on Redis Stream is implemented.

[0167] In one embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned data processing method based on Redis Stream when executed by a processor.

[0168] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0169] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method based on Redis Stream, characterized in that: The method comprises: Define a namespace, topic name, and configuration items, and create a topic information key in Redis according to the namespace, topic name, and configuration items. The topic information key is used to store topic configuration information. Receive the data written by the producer to the topic, extract the partition key passed in by the producer, calculate the partition ID corresponding to the written data according to the topic configuration information and the partition key, and send the written data to the Stream corresponding to the partition ID; Define a consumer group name, and create a consumer group information key in Redis according to the namespace, the topic name, and the consumer group name. The consumer group information key is used to store a consumer list and a partition allocation scheme. In response to a consumer change event, a new partition allocation scheme is generated according to the consumer group information key and using a partition rebalancing mechanism; Based on the new partition allocation scheme, update the partition locks corresponding to different consumers, and control consumers to consume the Stream data of the corresponding partitions according to the partition locks corresponding to the consumers.

2. The method according to claim 1, characterized in that The responding to the consumer change event and generating a new partition allocation scheme according to the consumer group information key and using the partition rebalancing mechanism includes: When a consumer joins, check whether the topic corresponding to the consumer to be added exists; If it exists, obtain the consumer group management lock corresponding to the consumer to be added; Generate a new partition allocation scheme according to the consumer group information key and using a partition rebalancing mechanism, and update the consumer group information key according to the new partition allocation scheme; Start the associated subtask corresponding to the consumer to be added, and send control information of the newly added consumer; Release the consumer group management lock.

3. The method according to claim 1, characterized in that The responding to the consumer change event and generating a new partition allocation scheme according to the consumer group information key and using the partition rebalancing mechanism includes: When a consumer leaves, check whether the topic corresponding to the departing consumer exists; If it exists, obtain the consumer group management lock corresponding to the leaving consumer; Close the associated subtask corresponding to the leaving consumer; Generate a new partition allocation scheme according to the consumer group information key and using a partition rebalancing mechanism, and update the consumer group information key according to the new partition allocation scheme; Send control information to remove the leaving consumer and release the consumer group management lock.

4. The method according to claim 2 or 3, characterized in that: The associated subtasks include heartbeat lock, periodic check of crashed consumers, and subscription control flow.

5. The method according to claim 1, characterized in that The responding to the consumer change event and generating a new partition allocation scheme according to the consumer group information key and using a partition rebalancing mechanism includes: In response to a consumer change event, identifying a first informed person corresponding to the consumer change event; The first insider obtains a list of currently active consumers from the consumer group information key; Re-allocating the mapping relationship between consumers and partitions according to the currently active consumer list and a preset allocation strategy, wherein the preset allocation strategy is generated based on load balancing and / or reducing excessive switching of consumers to partitions; The first informed party updates the mapping relationship between the reallocated consumers and partitions into the consumer group information key.

6. The method according to claim 5, characterized in that The reallocation of the mapping relationship between consumers and partitions according to the currently active consumer list and the preset allocation strategy includes: According to the currently active consumer list, obtain the number of currently active consumers N and the number of partitions M; According to the current number of active consumers N and the number of partitions M, determine the maximum number of partitions to be evenly distributed to each consumer; Consumers are sorted according to the time they joined the consumer group. Based on the sorting results and the maximum number of partitions allocated to each consumer, the mapping relationship between consumers and partitions is reallocated using a continuous partition allocation method.

7. The method according to claim 6, characterized in that The responding to the consumer change event and identifying the first informed party corresponding to the consumer change event comprises: Respond to consumer change events and identify the corresponding types of consumer change events; If the corresponding type of the consumer change event is consumer joining, the newly joined consumer is determined to be the first informed person; If the corresponding type of the consumer change event is consumer leaving, the departing consumer is determined to be the first informed person; If the corresponding type of the consumer change event is consumer crash, the first consumer who scans and finds the crashed consumer is determined to be the first informed party.

8. A data processing device based on Redis Stream, characterized in that: The device comprises: A topic creation module is used to define a namespace, a topic name, and configuration items, and to create a topic information key in Redis according to the namespace, the topic name, and configuration items. The topic information key is used to store topic configuration information. The data writing module is used to receive the data written by the producer to the topic, extract the partition key passed in by the producer, calculate the partition ID corresponding to the written data according to the topic configuration information and the partition key, and send the written data to the Stream corresponding to the partition ID; A consumer group creation module, used to define a consumer group name, and create a consumer group information key in Redis according to the namespace, the topic name, and the consumer group name, wherein the consumer group information key is used to store a consumer list and a partition allocation scheme; A partition module is used to respond to consumer change events and generate a new partition allocation plan based on the consumer group information key and using a partition rebalancing mechanism; The control module is used to update the partition locks corresponding to different consumers based on the new partition allocation scheme, and control consumers to consume the Stream data of the corresponding partitions according to the partition locks corresponding to the consumers.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images