Consumer offset information modification method and device, equipment, medium and product

Abstract

The invention relates to the technical field of data processing, and provides a consumer offset information modification method and device, equipment, a medium and a product. For each message in the main message middleware cluster, establishing an upstream and downstream corresponding relationship according to a theme, a partition, an upstream offset and a downstream offset corresponding to the message, and storing the upstream and downstream corresponding relationship to a database server; determining a first offset mapping relation corresponding to the consumer group according to first consumer offset information stored in the main message middleware cluster, and sending the first offset mapping relation to a database server, so that the database server determines a second offset mapping relation according to the first offset mapping relation and the upstream and downstream corresponding relation; and according to the second offset mapping relation corresponding to each consumer group sent by the database server, performing modification processing on second consumer offset information stored in the standby message middleware cluster. The accuracy of the second consumer offset information in the standby message-oriented middleware cluster can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to methods, devices, equipment, media and products for modifying consumer offset information. Background Art

[0002] The second consumer offset information stored in the standby message middleware needs to accurately reflect the progress of the consumer group's message consumption, so that when a cluster switch occurs, the standby message middleware cluster can directly control the consumer's message consumption based on the second consumer offset information.

[0003] In the related art, a computing device running a data migration tool can establish an upstream-downstream correspondence based on the upstream offset of a message in a primary message middleware cluster and the downstream offset of the message in a backup message middleware cluster, and store the upstream-downstream correspondence in a local memory. The computing device can synchronously modify the second consumer offset information in the backup message middleware cluster based on the first consumer offset information in the primary message middleware cluster and the upstream-downstream correspondence.

[0004] However, the method in the related art is to establish an upstream and downstream correspondence between messages at an interval of at least one message, which leads to the problem that the computing device in the related art cannot accurately modify the second consumer offset information in the backup message middleware cluster, which in turn leads to the problem that the backup message middleware cluster controls consumers to repeatedly consume messages. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device, medium and product for modifying consumer offset information, which can accurately modify the second consumer offset information in a backup message middleware cluster, thereby improving the accuracy of the second consumer offset information in the backup message middleware cluster, and thereby improving the accuracy of the backup message middleware cluster in controlling consumer consumption messages.

[0006] In a first aspect, an embodiment of the present application provides a method for modifying consumer offset information, including:

[0007] For each message in the main message middleware cluster, obtain the topic, partition, upstream offset, and downstream offset corresponding to the message;

[0008] Establish an upstream-downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message, and send the upstream-downstream correspondence relationship to the database server so that the database server stores and processes the upstream-downstream correspondence relationship;

[0009] According to the first consumer offset information stored in the main message middleware cluster, determine the first offset mapping relationship corresponding to each consumer group; the first offset mapping relationship includes a topic, a partition, and an upstream offset;

[0010] Sending a first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine a second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships; the second offset mapping relationship includes a topic, a partition, and a downstream offset;

[0011] Get the second offset mapping relationship corresponding to each consumer group sent by the database server;

[0012] According to the second offset mapping relationship corresponding to each consumer group, the second consumer offset information stored in the standby message middleware cluster is modified.

[0013] In one implementation, an upstream-downstream correspondence is established according to an upstream offset, a downstream offset, a topic, and a partition corresponding to a message, and the upstream-downstream correspondence is sent to a database server, including:

[0014] Establish upstream and downstream correspondence based on the upstream offset, downstream offset, topic, and partition corresponding to the message;

[0015] Store the upstream and downstream correspondence in local memory;

[0016] When all upstream and downstream correspondences stored in the local memory are identified and meet preset conditions, all upstream and downstream correspondences stored in the local memory are sent to the database server.

[0017] In one implementation, when all upstream and downstream correspondences stored in the local memory are identified and a preset condition is met, all upstream and downstream correspondences stored in the local memory are sent to the database server, including:

[0018] When it is identified that the number of all upstream and downstream correspondences stored in the local memory is greater than a preset threshold, all upstream and downstream correspondences stored in the local memory are sent to the database server.

[0019] In one implementation, determining the first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster includes:

[0020] For each consumer group, determine the topic corresponding to the consumer group and the partitions included in the topic;

[0021] The upstream offset is determined according to the consumer group, partition, topic, and first consumer offset information stored in the main message middleware cluster, and the first offset mapping relationship is determined according to the consumer group, partition, topic, and upstream offset.

[0022] In one implementation,

[0023] The database server stores multiple ordered sets; the key of each ordered set is the topic and partition; the score of each element in the ordered set is the upstream offset, and the value of each element is the downstream offset;

[0024] The topic and partition in the first offset mapping corresponding to the consumer group have the same key as the target sorted set in the multiple sorted sets;

[0025] The downstream offset in the first offset mapping relationship is the same as the value of the target element in the target ordered set;

[0026] The key of the target ordered set is the same as the topic and partition in the second offset mapping corresponding to the consumer group;

[0027] The score of the target element, which is the same as the upstream offset in the second offset mapping.

[0028] In one implementation,

[0029] The first offset mapping relationship is also used for the database server to delete the target upstream and downstream correspondence relationship according to the second offset mapping relationship corresponding to the consumer group; wherein the topic and partition in the target upstream and downstream correspondence relationship are the same as the topic and partition in the second offset mapping relationship, and the downstream offset in the target upstream and downstream correspondence relationship is smaller than the downstream offset in the second offset mapping relationship.

[0030] In a second aspect, an embodiment of the present application provides a device for modifying consumer offset information, including:

[0031] The acquisition module is used to obtain the topic, partition, upstream offset, and downstream offset corresponding to each message in the main message middleware cluster;

[0032] A processing module, used to establish an upstream and downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message;

[0033] The transceiver module is used to send the upstream and downstream correspondence relationship to the database server so that the database server can store and process the upstream and downstream correspondence relationship;

[0034] The processing module is further used to determine the first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster; the first offset mapping relationship includes a topic, a partition, and an upstream offset;

[0035] The transceiver module is also used to send the first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine the second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships; the second offset mapping relationship includes a topic, a partition, and a downstream offset;

[0036] The transceiver module is also used to obtain the second offset mapping relationship corresponding to each consumer group sent by the database server;

[0037] The processing module is also used to modify the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0038] In one implementation,

[0039] The processing module is specifically used to establish an upstream and downstream correspondence relationship according to the upstream offset, downstream offset, topic, and partition corresponding to the message;

[0040] The processing module is specifically used to store the upstream and downstream correspondence relationships in the local memory;

[0041] The transceiver module is specifically used to send all the upstream and downstream corresponding relationships stored in the local memory to the database server when all the upstream and downstream corresponding relationships stored in the local memory are identified and meet the preset conditions.

[0042] In one implementation, the transceiver module is specifically configured to:

[0043] When it is identified that the number of all upstream and downstream correspondences stored in the local memory is greater than a preset threshold, all upstream and downstream correspondences stored in the local memory are sent to the database server.

[0044] In one implementation, the processing module is specifically configured to:

[0045] For each consumer group, determine the topic corresponding to the consumer group and the partitions included in the topic;

[0046] The upstream offset is determined according to the consumer group, partition, topic, and first consumer offset information stored in the main message middleware cluster, and the first offset mapping relationship is determined according to the consumer group, partition, topic, and upstream offset.

[0047] In one implementation,

[0048] The database server stores multiple ordered sets; the key of each ordered set is the topic and partition; the score of each element in the ordered set is the upstream offset, and the value of each element is the downstream offset;

[0049] The topic and partition in the first offset mapping corresponding to the consumer group have the same key as the target sorted set in the multiple sorted sets;

[0050] The downstream offset in the first offset mapping relationship is the same as the value of the target element in the target ordered set;

[0051] The key of the target ordered set is the same as the topic and partition in the second offset mapping corresponding to the consumer group;

[0052] The score of the target element, which is the same as the upstream offset in the second offset mapping.

[0053] In one implementation,

[0054] The first offset mapping relationship is also used for the database server to delete the target upstream and downstream correspondence relationship according to the second offset mapping relationship corresponding to the consumer group; wherein the topic and partition in the target upstream and downstream correspondence relationship are the same as the topic and partition in the second offset mapping relationship, and the downstream offset in the target upstream and downstream correspondence relationship is smaller than the downstream offset in the second offset mapping relationship.

[0055] In a third aspect, an embodiment of the present application provides a computing device, including:

[0056] A processor, and a memory communicatively connected to the processor;

[0057] Memory is used to store computer executable instructions;

[0058] The processor is used to execute the computer-executable instructions stored in the memory to implement the method of the first aspect.

[0059] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method of the first aspect.

[0060] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which is used to implement the method of the first aspect when the computer program is executed by a processor.

[0061] The embodiment of the present application provides a method, device, equipment, medium and product for modifying consumer offset information. In the method, for each message in the main message middleware cluster, the computing device can obtain the topic, partition, upstream offset, and downstream offset corresponding to the message. The computing device can establish an upstream and downstream correspondence according to the topic, partition, upstream offset, and downstream offset corresponding to the message, and store the upstream and downstream correspondence to the database server. The computing device can determine the first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster; the first offset mapping relationship includes the topic, partition, and upstream offset. The computing device can send the first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine the second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all the upstream and downstream correspondences; the second offset mapping relationship includes the topic, partition, and downstream offset. The computing device can obtain the second offset mapping relationship corresponding to each consumer group sent by the database server, and modify the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group. The method of the embodiment of the present application can establish an upstream and downstream correspondence relationship for each message, so that the computing device can accurately determine the second offset mapping relationship (including the downstream offset) corresponding to each consumer group, and then the computing device can accurately modify the second consumer offset information in the standby message middleware cluster, thereby improving the accuracy of the second consumer offset information in the standby message middleware cluster, and then improving the accuracy of the standby message middleware cluster in controlling consumer consumption messages. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0063] Figure 1 A schematic diagram of an application scenario applicable to an embodiment of the present application;

[0064] Figure 2a A flowchart of a first embodiment of a method for modifying consumer offset information provided in an embodiment of the present application;

[0065] Figure 2b A schematic diagram of an ordered set provided in an embodiment of the present application;

[0066] Figure 3 A flowchart of a second embodiment of a method for modifying consumer offset information provided in an embodiment of the present application;

[0067] Figure 4 A flowchart of a third embodiment of a method for modifying consumer offset information provided in an embodiment of the present application;

[0068] Figure 5 A schematic diagram of the structure of a device for modifying consumer offset information provided in an embodiment of the present application;

[0069] Figure 6 A structural diagram of a computing device provided for this application. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments made by ordinary technicians in this field under the enlightenment of the embodiments belong to the scope of protection of the present application.

[0071] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0072] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0073] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0074] Glossary:

[0075] Topic: A topic is a logical unit used to classify and organize messages in a message middleware (such as Kafka). Each topic can be considered a message category or message stream. Producers can publish messages to a topic, and consumers can subscribe to and consume messages from a topic.

[0076] Partition: It is a physical sharding of a topic. Each topic can be divided into at least one partition, and each partition is an ordered, immutable sequence of messages. Partitions enable Kafka to process messages in parallel. Each partition can be processed by different consumers, thereby improving the throughput of the system. Through partitions, the message middleware can distribute the load to multiple brokers. Within a single partition, messages are ordered.

[0077] The second consumer offset information stored in the backup message middleware needs to accurately reflect the progress of the consumer group's message consumption, so that when a cluster switch occurs (from the main message middleware cluster to the backup message middleware cluster), the backup message middleware cluster can directly control the consumer (Consumer) to consume messages based on the second consumer offset information.

[0078] In the related art, a computing device running a data migration tool can establish an upstream-downstream correspondence based on the upstream offset of a message in a primary message middleware cluster and the downstream offset of the message in a backup message middleware cluster, and store the upstream-downstream correspondence in a local memory. The computing device can synchronously modify the second consumer offset information in the backup message middleware cluster based on the first consumer offset information in the primary message middleware cluster and the upstream-downstream correspondence.

[0079] However, the method in the related art is to establish an upstream and downstream correspondence between messages at an interval of at least one message, which leads to the problem that the computing device in the related art cannot accurately modify the second consumer offset information in the backup message middleware cluster, which in turn leads to the problem that the backup message middleware cluster controls consumers to repeatedly consume messages.

[0080] Based on the above technical problems, an embodiment of the present application provides a method for modifying consumer offset information. In this method, for each message in the main message middleware cluster, the computing device can obtain the topic, partition, upstream offset, and downstream offset corresponding to the message. The computing device can establish an upstream and downstream correspondence according to the topic, partition, upstream offset, and downstream offset corresponding to the message, and store the upstream and downstream correspondence to the database server. The computing device can determine the first offset mapping relationship (including topic, partition, and upstream offset) corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster. The computing device can send the first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine the second offset mapping relationship (including topic, partition, and downstream offset) corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream correspondences. The computing device can obtain the second offset mapping relationship corresponding to each consumer group sent by the database server, and modify the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0081] The method of the embodiment of the present application can establish an upstream and downstream correspondence relationship for each message, so that the computing device can accurately determine the second offset mapping relationship (including the downstream offset) corresponding to each consumer group, and then the computing device can accurately modify the second consumer offset information in the standby message middleware cluster, thereby improving the accuracy of the second consumer offset information in the standby message middleware cluster, and then improving the accuracy of the standby message middleware cluster in controlling consumer consumption messages.

[0082] For ease of understanding, the following Figure 1 The examples are used to illustrate the application scenarios to which the embodiments of the present application are applicable.

[0083] For example, Figure 1 Schematic diagram of an application scenario applicable to the embodiment of the present application. Figure 1 As shown, the application scenario includes: a main message middleware cluster 10 , a computing device 20 , a backup message middleware cluster 30 and a database server 40 .

[0084] The computing device 20 is communicatively connected to the primary message middleware cluster 10 , the standby message middleware cluster 30 , and the database server 40 , respectively.

[0085] It should be noted that the computing device 20 can be a terminal device (such as a computer), a server, or a server cluster. The embodiment of the present application is not limited to this. A data migration tool (such as MirrorMaker) can be run on the computing device 20. It should be noted that when MirrorMaker is run on the computing device 20, MirrorMaker can include three components, namely, a first component (MirrorSourceConnector), a second component (MirrorCheckpointConnector), and a third component (MirrorHeartbeatConnector). MirrorMaker can perform data migration through three components.

[0086] It should also be noted that message middleware can be run on the main message middleware cluster 10 and the standby message middleware 30. Exemplarily, the message middleware can be Kafka, or other message middleware, which is not limited in the embodiment of the present application.

[0087] The embodiments of this application are not Figure 1 The actual form of the equipment included in the Figure 1 The interaction mode between devices is limited, and in the application of the solution, it can be set according to actual needs.

[0088] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0089] Figure 2a This is a flow chart of a first embodiment of a method for modifying consumer offset information provided in an embodiment of the present application. Figure 2a , the method specifically comprises the following steps:

[0090] S201: The computing device obtains the topic, partition, upstream offset, and downstream offset corresponding to each message in the main message middleware cluster.

[0091] In this embodiment, for each message in the main message middleware cluster, the computing device may obtain the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0092] In one implementation, during the data synchronization process between the primary message middleware cluster and the backup message middleware cluster, the computing device may pull messages from the primary message middleware cluster and send the messages to the backup message middleware cluster.

[0093] The computing device may record the position of the message in a partition of a topic of the main message middleware cluster, and determine the position as the upstream offset. In addition, the computing device may record the topic and partition where the message is located.

[0094] The computing device may record the position of the message in a partition of a topic in the backup message middleware cluster, and determine the position as the downstream offset. In addition, the computing device may record the topic and partition where the message is located. It should be noted that the topic and partition where the message is located in the primary message middleware cluster are the same as the topic and partition where the message is located in the backup message middleware cluster.

[0095] It should be noted that, in one implementation, the computing device may save the topic, partition, upstream offset, and downstream offset corresponding to the message in a topic of the main message middleware cluster - offset synchronization information (offset_sync).

[0096] It should be noted that, when the data migration tool running on the computing device is MirrorMaker, the first component (MirrorSourceConnector) of MirrorMaker can pull messages from the main message middleware cluster and send the information to the backup message middleware cluster. The first component (MirrorSourceConnector) obtains the topic, partition, upstream offset, and downstream offset corresponding to the message, and saves the topic, partition, upstream offset, and downstream offset corresponding to the message in the offset synchronization information (offset_sync) of the main message middleware cluster.

[0097] S202: The computing device establishes an upstream and downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0098] In this embodiment, the computing device may establish an upstream and downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0099] In one implementation, the computing device can pull the topic, partition, upstream offset, and downstream offset corresponding to the message from the offset synchronization information (offset_sync) of the main message middleware cluster. The computing device can establish an upstream and downstream correspondence (offsetMap) based on the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0100] It should be noted that, in one implementation, when the data migration tool running on the computing device is MirrorMaker, the first component (MirrorSourceConnector) of MirrorMaker can establish an upstream and downstream correspondence according to the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0101] S203: The computing device sends the upstream and downstream correspondence relationship to the database server.

[0102] In this embodiment, the computing device may send the upstream and downstream correspondence to the database server. It should be noted that, in one implementation, the database server may be a server running a remote dictionary service (Remote Dictionary Server, Redis for short). It should be noted that, from a software perspective, it is the MirrorSourceConnector running on the computing device that calls the lua script (a lightweight and efficient scripting language) in Redis to send the upstream and downstream correspondence to Redis.

[0103] In one implementation:

[0104] The computing device may store the upstream and downstream correspondence relationships in the local memory of the computing device.

[0105] The computing device may send all the upstream and downstream correspondences stored in the local memory to the database server when it identifies all the upstream and downstream correspondences stored in the local memory and meets the preset conditions. In one implementation, the computing device may send all the upstream and downstream correspondences stored in the local memory to the database server when it identifies that the number of all the upstream and downstream correspondences stored in the local memory is greater than a preset threshold.

[0106] S204: The database server stores and processes the upstream and downstream correspondence relationships.

[0107] In this embodiment, the database server may store and process the upstream and downstream corresponding relationships after acquiring the upstream and downstream corresponding relationships sent by the computing device.

[0108] In one implementation, when the database server is a server running Redis, the database may use an ordered set to store the upstream and downstream correspondence relationship.

[0109] The database server stores multiple ordered sets. Figure 2b A schematic diagram of an ordered set provided in an embodiment of the present application. For example, Figure 2b Two ordered sets are shown.

[0110] The key of an ordered set is the topic and partition. An ordered set includes multiple elements, where the score of an element is the upstream offset in the upstream-downstream correspondence, and the value of an element is the downstream offset in the upstream-downstream correspondence.

[0111] It should be noted that the database server may arrange multiple elements in an ordered set in ascending order according to the score of each element in the ordered set.

[0112] S205: The computing device determines a first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster.

[0113] In this embodiment, the computing device can determine the first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster, wherein the first offset mapping relationship includes a topic, a partition, and an upstream offset.

[0114] It should be noted that topics and partitions can be represented by "topicAndPartition".

[0115] It should also be noted that each consumer group can subscribe to at least one topic, and each topic can include at least one partition, that is, each consumer group can consume messages in at least one partition of at least one topic. Therefore, each consumer group can correspond to at least one first offset mapping relationship.

[0116] In one implementation:

[0117] For each consumer group, the computing device can determine the topic corresponding to the consumer group and the partitions included in the topic. It should be noted that the topic corresponding to the consumer group refers to the topic subscribed by the consumer group. The topic can include partitions, which are used to store messages. It should also be noted that the consumer group can correspond to at least one topic, and each topic can include at least one partition.

[0118] The computing device can determine the upstream offset based on the consumer group, partition, topic, and the first consumer offset information stored in the main message cluster. It should be noted that the first consumer offset information (_consumer_offsets) is stored in the main message middleware cluster. Among them, the first consumer offset information records the upstream offset that has a corresponding relationship with the consumer group, topic, and partition. Among them, the upstream offset reflects the progress of the consumer's consumption information in the partition of the topic.

[0119] The computing device may determine a first offset mapping relationship according to the consumer group, the partition, the topic, and the upstream offset.

[0120] After determining the first offset mapping relationship, the computing device may store the first offset mapping relationship in a local memory.

[0121] It should be noted that the first offset mapping relationship is stored in the offsetMap structure. It should also be noted that the topic and partition in the first offset mapping relationship can be represented by "topicAndPartition".

[0122] It should also be noted that, in one implementation, when the data migration tool running on the computing device is MirrorMaker, the second component of MirrorMaker (MirrorCheckpointConnector) determines the first offset mapping relationship corresponding to each consumer group based on the first consumer offset information stored in the main message middleware cluster.

[0123] S206: The computing device sends the first offset mapping relationship corresponding to each consumer group to the database server.

[0124] In this embodiment, after determining the first offset mapping relationships corresponding to all consumer groups, the computing device may send the first offset mapping relationship corresponding to each consumer group to the database server.

[0125] It should be noted that, based on at least one first offset mapping relationship corresponding to the consumer group, the computing device can send at least one first offset mapping relationship corresponding to all consumer groups to the database server after determining at least one first offset mapping relationship corresponding to all consumer groups.

[0126] S207: The database server determines a second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships.

[0127] In this embodiment, the database server may determine the second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships.

[0128] In one implementation, when the database server is a Redis server, the database server stores multiple ordered sets, wherein the key of each ordered set is a topic and a partition (a topic identifier and a partition identifier); the score of each element in the ordered set is an upstream offset, and the value of each element is a downstream offset.

[0129] The first offset mapping relationship includes a topic, a partition (a partition included in the topic), and an upstream offset (the upstream offset reflects the consumption progress of the consumer group in the partition of the topic).

[0130] For each first offset mapping relationship, the database server can determine, among multiple ordered sets, an ordered set whose key is the same as the topic and partition in the first offset mapping relationship as the target ordered set. The database server can determine, in the target ordered set, an element whose score is the same as the upstream offset in the first offset mapping relationship as the target element. The database server can determine that the value of the target element is the downstream offset in the second offset mapping relationship. In one implementation, the database server can use the "zrangebyscore topic-partition offsetoffset" command to determine the downstream offset. It should be noted that "topic-partition" refers to the topic and partition, and in "offset offset", the first offset and the second offset are both upstream offsets in the first offset mapping relationship.

[0131] In addition, the database server may also determine the subject and partition in the first offset mapping relationship as the subject and partition in the second offset mapping relationship.

[0132] Through the above method, the second offset mapping relationship corresponding to the consumer group can be determined.

[0133] It should be noted that the topic and partition in the first offset mapping relationship corresponding to the consumer group are the same as the key of the target ordered set in the multiple ordered sets. The downstream offset in the first offset mapping relationship is the same as the value of the target element in the target ordered set. The key of the target ordered set is the same as the topic and partition in the second offset mapping relationship corresponding to the consumer group. The score of the target element is the same as the upstream offset in the second offset mapping relationship.

[0134] It can be understood that the topic and partition in the second offset mapping relationship corresponding to a consumer group are the same as the topic and partition in the first offset mapping relationship corresponding to the consumer group. The downstream offset in the second offset mapping relationship corresponding to the consumer group is the downstream offset of a message, and the upstream offset in the first offset mapping relationship corresponding to the consumer group is the upstream offset of the message.

[0135] That is to say, the second offset mapping relationship corresponding to a consumer group corresponds to the first offset mapping relationship corresponding to the consumer group. It should also be noted that when a consumer group corresponds to N (N is a positive integer greater than or equal to 1) first offset mapping relationships, the consumer group corresponds to N second offset mapping relationships, and each first offset mapping relationship corresponds to one second offset mapping relationship.

[0136] In addition, it should be noted that the second offset mapping relationship is stored in the resultMap structure.

[0137] S208: The computing device obtains the second offset mapping relationship corresponding to each consumer group sent by the database server.

[0138] In this embodiment, the computing device may obtain the second offset mapping relationship corresponding to each consumer group sent by the database server. In one implementation, when each consumer group corresponds to at least one second offset mapping relationship, the computing device may obtain at least one second offset mapping relationship corresponding to each consumer group sent by the database server.

[0139] It can be understood that in one implementation, the computing device can obtain the resultMap structure (storing the second offset mapping relationship corresponding to each consumer group) sent by the database server.

[0140] S209: The computing device modifies the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0141] In this embodiment, the standby message middleware cluster stores the second consumer offset information (_consumer_offset), wherein the second consumer offset information includes the topic, partition, and downstream offset corresponding to the consumer group.

[0142] The computing device can modify the second consumer offset information stored in the backup message middleware cluster according to the second offset mapping relationship corresponding to each consumer group, so that when the main message middleware cluster and the backup message middleware cluster switch, the backup message middleware cluster can directly control the consumer (Consumer) to consume messages based on the second consumer offset information, avoiding the problem of duplicate consumption and improving the accuracy of consumed messages.

[0143] Beneficial effects of this embodiment: For each message in the main message middleware cluster, the computing device can obtain the topic, partition, upstream offset, and downstream offset corresponding to the message. The computing device can establish an upstream and downstream correspondence according to the topic, partition, upstream offset, and downstream offset corresponding to the message, and store the upstream and downstream correspondence to the database server. The computing device can determine the first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster; the first offset mapping relationship includes the topic, partition, and upstream offset. The computing device can send the first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine the second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all the upstream and downstream correspondences; the second offset mapping relationship includes the topic, partition, and downstream offset. The computing device can obtain the second offset mapping relationship corresponding to each consumer group sent by the database server, and modify the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group. The method of the embodiment of the present application can establish an upstream and downstream correspondence relationship for each message, so that the computing device can accurately determine the second offset mapping relationship (including the downstream offset) corresponding to each consumer group, and then the computing device can accurately modify the second consumer offset information in the standby message middleware cluster, thereby improving the accuracy of the second consumer offset information in the standby message middleware cluster, and then improving the accuracy of the standby message middleware cluster in controlling consumer consumption messages.

[0144] Figure 3 This is a flow chart of a second embodiment of a method for modifying consumer offset information provided in an embodiment of the present application. Figure 3 , the method specifically comprises the following steps:

[0145] S301: The computing device obtains the topic, partition, upstream offset, and downstream offset corresponding to each message in the main message middleware cluster.

[0146] In this embodiment, for each message in the main message middleware cluster, the computing device may obtain the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0147] The specific implementation process is the same as S201 and will not be repeated here.

[0148] S302: The computing device establishes an upstream and downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0149] In this embodiment, the computing device may establish an upstream and downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0150] The specific implementation process is the same as S202 and will not be repeated here.

[0151] S303: The computing device stores the upstream and downstream correspondence relationship in the local memory.

[0152] In this embodiment, after establishing the upstream and downstream correspondence relationship corresponding to each message, the computing device may store the upstream and downstream correspondence relationship in the local memory.

[0153] S304: When the computing device identifies all upstream and downstream correspondences stored in the local memory and meets preset conditions, it sends all upstream and downstream correspondences stored in the local memory to the database server.

[0154] In this embodiment, the computing device can send all upstream and downstream correspondences stored in the local memory to the database server when identifying all upstream and downstream correspondences stored in the local memory and satisfying preset conditions.

[0155] In one implementation, the computing device may send all upstream and downstream correspondences stored in the local memory to the database server when it is identified that the number of all upstream and downstream correspondences stored in the local memory is greater than a preset threshold. For example, the preset threshold may be 1000.

[0156] In one implementation, the computing device may send all upstream and downstream correspondences stored in the local memory to the database server upon recognizing that the number of all upstream and downstream correspondences stored in the local memory is greater than a preset threshold and the memory occupancy rate of the local memory is greater than a preset occupancy rate.

[0157] S305: The database server stores and processes the upstream and downstream correspondence relationship.

[0158] In this embodiment, the database server may store and process the upstream and downstream corresponding relationships after acquiring the upstream and downstream corresponding relationships sent by the computing device.

[0159] The specific implementation process is the same as S204 and will not be repeated here.

[0160] S306: The computing device determines a first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster.

[0161] In this embodiment, the computing device can determine the first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster, wherein the first offset mapping relationship includes a topic, a partition, and an upstream offset.

[0162] The specific implementation process is the same as S205 and will not be repeated here.

[0163] S307: The computing device sends the first offset mapping relationship corresponding to each consumer group to the database server.

[0164] In this embodiment, the computing device may send the first offset mapping relationship corresponding to each consumer group to the database server.

[0165] The specific implementation process is the same as S206 and will not be repeated here.

[0166] S308: The database server determines a second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships.

[0167] In this embodiment, the database server may determine the second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships.

[0168] The specific implementation process is the same as S207 and will not be repeated here.

[0169] S309: The computing device obtains the second offset mapping relationship corresponding to each consumer group sent by the database server.

[0170] In this embodiment, the computing device may obtain the second offset mapping relationship corresponding to each consumer group sent by the database server.

[0171] The specific implementation process is the same as S208 and will not be repeated here.

[0172] S310: The computing device modifies the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0173] In this embodiment, the computing device may modify the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0174] The specific implementation process is the same as S209 and will not be repeated here.

[0175] Beneficial effects of this embodiment: The computing device can establish upstream and downstream correspondences according to the upstream offset, downstream offset, topic, and partition corresponding to the message, and store the upstream and downstream correspondences in the local memory. The computing device can send all the upstream and downstream correspondences stored in the local memory to the database server when it identifies all the upstream and downstream correspondences stored in the local memory and meets the preset conditions. In the above manner, the number of interactions between the computing device and the database server can be reduced, the communication overhead between the computing device and the database server can be saved, and the storage efficiency of the upstream and downstream correspondences can be improved.

[0176] Figure 4 This is a flow chart of a third embodiment of a method for modifying consumer offset information provided in an embodiment of the present application. Figure 4 , the method specifically comprises the following steps:

[0177] S401: The computing device obtains the topic, partition, upstream offset, and downstream offset corresponding to each message in the main message middleware cluster.

[0178] In this embodiment, for each message in the main message middleware cluster, the computing device may obtain the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0179] The specific implementation process is the same as S201 and will not be repeated here.

[0180] S402: The computing device establishes an upstream and downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0181] In this embodiment, the computing device may establish an upstream and downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0182] The specific implementation process is the same as S202 and will not be repeated here.

[0183] S403: The computing device stores the upstream and downstream correspondence relationship in the local memory.

[0184] In this embodiment, after establishing the upstream and downstream correspondence relationship corresponding to each message, the computing device may store the upstream and downstream correspondence relationship in the local memory.

[0185] S404: When the computing device identifies all upstream and downstream correspondences stored in the local memory and meets preset conditions, it sends all upstream and downstream correspondences stored in the local memory to the database server.

[0186] In this embodiment, the computing device can send all upstream and downstream correspondences stored in the local memory to the database server when identifying all upstream and downstream correspondences stored in the local memory and satisfying preset conditions.

[0187] The specific implementation process is the same as S304 and will not be repeated here.

[0188] S405: The database server stores and processes the upstream and downstream correspondence relationship.

[0189] In this embodiment, the database server may store and process the upstream and downstream corresponding relationships after acquiring the upstream and downstream corresponding relationships sent by the computing device.

[0190] The specific implementation process is the same as S204 and will not be repeated here.

[0191] S406: The computing device determines a first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster.

[0192] In this embodiment, the computing device can determine the first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster, wherein the first offset mapping relationship includes a topic, a partition, and an upstream offset.

[0193] The specific implementation process is the same as S205 and will not be repeated here.

[0194] S407: The computing device sends the first offset mapping relationship corresponding to each consumer group to the database server.

[0195] In this embodiment, the computing device may send the first offset mapping relationship corresponding to each consumer group to the database server.

[0196] The specific implementation process is the same as S206 and will not be repeated here.

[0197] S408: The database server determines a second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships.

[0198] In this embodiment, the database server may determine the second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships.

[0199] The specific implementation process is the same as S207 and will not be repeated here.

[0200] S409: The database server deletes the target upstream and downstream correspondence relationship according to the second offset mapping relationship corresponding to the consumer group.

[0201] In this embodiment, after determining the second offset mapping relationship corresponding to the consumer group, the database server may delete the target upstream and downstream corresponding relationship according to the second offset mapping relationship.

[0202] Among them, the topic and partition in the target upstream and downstream correspondence relationship are the same as the topic and partition in the second offset mapping relationship, and the downstream offset in the target upstream and downstream correspondence relationship is smaller than the downstream offset in the second offset mapping relationship.

[0203] In one implementation, the database server may use the "zremrangebyscore 0 offset" command to delete the target upstream and downstream correspondence relationship. It should be noted that "offset" refers to the downstream offset in the second offset mapping relationship.

[0204] It should be noted that, in one implementation, after determining the second offset mapping relationship corresponding to a consumer group, the database server can delete at least one target upstream and downstream correspondence relationship according to the second offset mapping relationship. In one implementation, after determining all second offset mapping relationships corresponding to all consumer groups, the database server can delete all target upstream and downstream correspondence relationships according to all second offset mapping relationships corresponding to all consumers.

[0205] S410: The computing device obtains the second offset mapping relationship corresponding to each consumer group sent by the database server.

[0206] In this embodiment, the computing device may obtain the second offset mapping relationship corresponding to each consumer group sent by the database server.

[0207] The specific implementation process is the same as S208 and will not be repeated here.

[0208] S411: The computing device modifies the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0209] In this embodiment, the computing device may modify the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0210] The specific implementation process is the same as S209 and will not be repeated here.

[0211] Beneficial effects of this embodiment: In this embodiment, after determining a second offset mapping relationship corresponding to a consumer group, the database server can delete the target upstream and downstream correspondence relationship according to the second offset mapping relationship. Among them, the topic and partition in the target upstream and downstream correspondence relationship are the same as the topic and partition in the second offset mapping relationship, and the downstream offset in the target upstream and downstream correspondence relationship is smaller than the downstream offset in the second offset mapping relationship. Through the above method, the upstream and downstream correspondences corresponding to the messages that have been consumed in the database server can be deleted in time, avoiding the database server from wasting space to store the target upstream and downstream correspondences, which in turn causes the storage space of the database server to be filled up. In other words, through the above method, the utilization rate of the memory space of the database server is improved.

[0212] Figure 5 This is a schematic diagram of the structure of a device for modifying consumer offset information provided by an embodiment of the present application. Figure 5 As shown, the device 50 for modifying consumer offset information includes: an acquisition module 51, a processing module 52, and a transceiver module 53.

[0213] The acquisition module 51 is used to acquire the topic, partition, upstream offset, and downstream offset corresponding to each message in the main message middleware cluster;

[0214] The processing module 52 is used to establish an upstream and downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message;

[0215] The transceiver module 53 is used to send the upstream and downstream correspondence relationship to the database server so that the database server can store and process the upstream and downstream correspondence relationship;

[0216] The processing module 52 is further used to determine the first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster; the first offset mapping relationship includes a topic, a partition, and an upstream offset;

[0217] The transceiver module 53 is also used to send the first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine the second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships; the second offset mapping relationship includes a topic, a partition, and a downstream offset;

[0218] The transceiver module 53 is also used to obtain the second offset mapping relationship corresponding to each consumer group sent by the database server;

[0219] The processing module 52 is further configured to modify the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0220] The device for modifying consumer offset information provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0221] In one implementation,

[0222] The processing module 52 is specifically used to establish an upstream and downstream correspondence relationship according to the upstream offset, downstream offset, topic, and partition corresponding to the message;

[0223] The processing module 52 is specifically used to store the upstream and downstream correspondence relationship in the local memory;

[0224] The transceiver module 53 is specifically used to send all the upstream and downstream corresponding relationships stored in the local memory to the database server when all the upstream and downstream corresponding relationships stored in the local memory are identified and meet the preset conditions.

[0225] The device for modifying consumer offset information provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0226] In one implementation, the transceiver module 53 is specifically configured to:

[0227] When it is identified that the number of all upstream and downstream correspondences stored in the local memory is greater than a preset threshold, all upstream and downstream correspondences stored in the local memory are sent to the database server.

[0228] The device for modifying consumer offset information provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0229] In one implementation, the processing module 52 is specifically configured to:

[0230] For each consumer group, determine the topic corresponding to the consumer group and the partitions included in the topic;

[0231] The upstream offset is determined according to the consumer group, partition, topic, and first consumer offset information stored in the main message middleware cluster, and the first offset mapping relationship is determined according to the consumer group, partition, topic, and upstream offset.

[0232] The device for modifying consumer offset information provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0233] In one implementation,

[0234] The database server stores multiple ordered sets; the key of each ordered set is the topic and partition; the score of each element in the ordered set is the upstream offset, and the value of each element is the downstream offset;

[0235] The topic and partition in the first offset mapping corresponding to the consumer group have the same key as the target sorted set in the multiple sorted sets;

[0236] The downstream offset in the first offset mapping relationship is the same as the value of the target element in the target ordered set;

[0237] The key of the target ordered set is the same as the topic and partition in the second offset mapping corresponding to the consumer group;

[0238] The score of the target element, which is the same as the upstream offset in the second offset mapping.

[0239] The device for modifying consumer offset information provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0240] In one implementation,

[0241] The first offset mapping relationship is also used for the database server to delete the target upstream and downstream correspondence relationship according to the second offset mapping relationship corresponding to the consumer group; wherein the topic and partition in the target upstream and downstream correspondence relationship are the same as the topic and partition in the second offset mapping relationship, and the downstream offset in the target upstream and downstream correspondence relationship is smaller than the downstream offset in the second offset mapping relationship.

[0242] The device for modifying consumer offset information provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0243] Figure 6 This is a structural diagram of a computing device provided by this application. Figure 6 As shown, the computing device 60 includes a processor 61 and a memory 62. The processor 61 is in communication connection with the memory 62, and the memory 62 is used to store computer-executable instructions; the processor 61 is configured to execute the technical solution in any of the above method embodiments by executing the computer-executable instructions stored in the memory 62.

[0244] Optionally, the memory 62 may be independent or integrated with the processor 61. Optionally, when the memory 62 is a device independent of the processor 61, the computing device 60 may further include: a bus 63 for connecting the above devices.

[0245] The computing device is used to execute the technical solution in any of the aforementioned method embodiments, and its implementation principle and technical effects are similar and will not be repeated here.

[0246] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the technical solution provided by any of the aforementioned method embodiments.

[0247] An embodiment of the present application also provides a computer program product, including a computer program, which is used to implement the technical solution provided by the aforementioned method embodiment when executed by a processor.

[0248] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0249] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for modifying consumer offset information, characterized in that: include: For each message in the main message middleware cluster, obtain the topic, partition, upstream offset, and downstream offset corresponding to the message; Establish an upstream-downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message, and send the upstream-downstream correspondence relationship to the database server so that the database server stores and processes the upstream-downstream correspondence relationship; Determine the first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster; the first offset mapping relationship includes a topic, a partition, and an upstream offset; Sending the first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine the second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream correspondences; the second offset mapping relationship includes a topic, a partition, and a downstream offset; Obtain a second offset mapping relationship corresponding to each consumer group sent by the database server; According to the second offset mapping relationship corresponding to each consumer group, the second consumer offset information stored in the standby message middleware cluster is modified.

2. The method according to claim 1, characterized in that The establishing an upstream and downstream correspondence relationship according to the upstream offset, downstream offset, topic, and partition corresponding to the message, and sending the upstream and downstream correspondence relationship to the database server includes: Establishing the upstream and downstream correspondence relationship according to the upstream offset, downstream offset, topic, and partition corresponding to the message; Storing the upstream and downstream correspondence relationship in local memory; When all the upstream and downstream correspondences stored in the local memory are identified and meet a preset condition, all the upstream and downstream correspondences stored in the local memory are sent to the database server.

3. The method according to claim 2, characterized in that The step of sending all the upstream and downstream corresponding relationships stored in the local memory to the database server when all the upstream and downstream corresponding relationships stored in the local memory are identified and meet a preset condition includes: When it is identified that the number of all the upstream and downstream correspondence relationships stored in the local memory is greater than a preset threshold, all the upstream and downstream correspondence relationships stored in the local memory are sent to the database server.

4. The method according to claim 1, characterized in that: The determining, according to the first consumer offset information stored in the main message middleware cluster, the first offset mapping relationship corresponding to each consumer group includes: For each consumer group, determine the topic corresponding to the consumer group and determine the partitions included in the topic; An upstream offset is determined according to the consumer group, the partition, the topic, and the first consumer offset information stored in the main message middleware cluster, and a first offset mapping relationship is determined according to the consumer group, the partition, the topic, and the upstream offset.

5. The method according to claim 4, characterized in that The database server stores multiple ordered sets; the key of each ordered set is a topic and a partition; the score of each element in the ordered set is an upstream offset, and the value of each element is a downstream offset; The topic and partition in the first offset mapping relationship corresponding to the consumer group are the same as the key of the target ordered set in the multiple ordered sets; The downstream offset in the first offset mapping relationship is the same as the value of the target element in the target ordered set; The key of the target ordered set is the same as the topic and partition in the second offset mapping relationship corresponding to the consumer group; The score of the target element is the same as the upstream offset in the second offset mapping relationship.

6. The method according to any one of claims 1 to 5, characterized in that: The first offset mapping relationship is also used for the database server to delete the target upstream and downstream correspondence relationship according to the second offset mapping relationship corresponding to the consumer group; wherein the topic and partition in the target upstream and downstream correspondence relationship are the same as the topic and partition in the second offset mapping relationship, and the downstream offset in the target upstream and downstream correspondence relationship is smaller than the downstream offset in the second offset mapping relationship.

7. A device for modifying consumer offset information, characterized in that: include: An acquisition module, used to acquire the topic, partition, upstream offset, and downstream offset corresponding to each message in the main message middleware cluster; A processing module, used to establish an upstream and downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message; A transceiver module, used for sending the upstream and downstream correspondence relationship to a database server, so that the database server stores and processes the upstream and downstream correspondence relationship; The processing module is further used to determine the first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster; the first offset mapping relationship includes a topic, a partition, and an upstream offset; The transceiver module is further used to send the first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine the second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships; the second offset mapping relationship includes a topic, a partition, and a downstream offset; The transceiver module is further used to obtain a second offset mapping relationship corresponding to each consumer group sent by the database server; The processing module is further used to modify the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

8. A computing device, characterized in that include: A processor, and a memory communicatively connected to the processor; The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program, which is used to implement the method according to any one of claims 1 to 6 when being executed by a processor.

Images