Method, apparatus, device, medium and product for modifying consumer offset information

Abstract

The application relates to the technical field of data processing, and provides a modification method and device for consumer offset information, equipment, a medium and a product. For each message in a main message middleware cluster, an upstream-downstream correspondence relationship is established according to a topic, a partition, an upstream offset and a downstream offset corresponding to the message, and the correspondence relationship is stored in a database server; a first offset mapping relationship corresponding to a consumer group is determined according to first consumer offset information stored by the main message middleware cluster, the first offset mapping relationship is sent to the database server, so that the database server determines a second offset mapping relationship according to the first offset mapping relationship and the upstream-downstream correspondence relationship; and second consumer offset information stored by a standby message middleware cluster is modified according to the second offset mapping relationship corresponding to each consumer group sent by the database server. The accuracy of the second consumer offset information in the standby message middleware cluster can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a consumer offset information modification method and device, equipment, medium and product. BACKGROUND

[0002] The second consumer offset information stored in the standby message middleware needs to accurately reflect the progress of the consumer group consuming messages, so that when cluster switching occurs, the standby message middleware cluster can directly control the consumer to consume messages based on the second consumer offset information.

[0003] In the related art, the computing device running the data migration tool can establish an upstream-downstream correspondence relationship based on the upstream offset of the message in the primary message middleware cluster and the downstream offset of the message in the standby message middleware cluster, and store the upstream-downstream correspondence relationship to the local memory. The computing device can synchronize and modify the second consumer offset information in the standby message middleware cluster based on the first consumer offset information in the primary message middleware cluster and the upstream-downstream correspondence relationship.

[0004] However, the method in the related art establishes the upstream-downstream correspondence relationship of one message at an interval of at least one message, which causes the method in the related art to have the problem that the computing device cannot accurately modify the second consumer offset information in the standby message middleware cluster, and further causes the method in the related art to have the problem that the standby message middleware cluster controls the consumer to repeatedly consume messages. SUMMARY

[0005] The consumer offset information modification method, device, equipment, medium and product provided by the embodiments of the present application can accurately modify the second consumer offset information in the standby message middleware cluster, improve the accuracy of the second consumer offset information in the standby message middleware cluster, and further improve the accuracy of the standby message middleware cluster in controlling the consumer to consume messages.

[0006] In a first aspect, the embodiments of the present application provide a consumer offset information modification method, comprising:

[0007] For each message in the primary message middleware cluster, obtaining the topic, partition, upstream offset, and downstream offset corresponding to the message;

[0008] According to the topic, partition, upstream offset, and downstream offset corresponding to the message, establishing an upstream-downstream correspondence relationship, and sending the upstream-downstream correspondence relationship to a database server to enable the database server to store and process the upstream-downstream correspondence relationship;

[0009] According to the first consumer offset information stored by the primary message middleware cluster, a first offset mapping relationship corresponding to each consumer group is determined; the first offset mapping relationship includes a topic, a partition, and an upstream offset;

[0010] The first offset mapping relationship corresponding to each consumer group is sent to the database server; the first offset mapping relationship is used for the database server to determine a second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships; the second offset mapping relationship includes a topic, a partition, and a downstream offset;

[0011] The second offset mapping relationship corresponding to each consumer group sent by the database server is obtained;

[0012] According to the second offset mapping relationship corresponding to each consumer group, the second consumer offset information stored by the standby message middleware cluster is modified.

[0013] In an implementation manner, the upstream and downstream corresponding relationships are established according to the upstream offset, the downstream offset, the topic, and the partition corresponding to the message, and the upstream and downstream corresponding relationships are sent to the database server, including:

[0014] The upstream and downstream corresponding relationships are established according to the upstream offset, the downstream offset, the topic, and the partition corresponding to the message;

[0015] The upstream and downstream corresponding relationships are stored in the local memory;

[0016] In a case where it is identified that all the upstream and downstream corresponding relationships stored in the local memory satisfy a preset condition, all the upstream and downstream corresponding relationships stored in the local memory are sent to the database server.

[0017] In an implementation manner, in a case where it is identified that all the upstream and downstream corresponding relationships stored in the local memory satisfy a preset condition, all the upstream and downstream corresponding relationships stored in the local memory are sent to the database server, including:

[0018] In a case where it is identified that the number of all the upstream and downstream corresponding relationships stored in the local memory is greater than a preset threshold, all the upstream and downstream corresponding relationships stored in the local memory are sent to the database server.

[0019] In an implementation manner, according to the first consumer offset information stored by the primary message middleware cluster, a first offset mapping relationship corresponding to each consumer group is determined, including:

[0020] For each consumer group, a topic corresponding to the consumer group is determined, and a partition included in the topic is determined;

[0021] According to the first consumer offset information of the consumer group, the partition, the topic, and the main message middleware cluster storage, an upstream offset is determined, and according to the consumer group, the partition, the topic, and the upstream offset, a first offset mapping relationship is determined.

[0022] In an implementation manner,

[0023] The database server stores a plurality of ordered sets; the key of each ordered set is a topic and a partition; the score of each element in the ordered set is an upstream offset, and the value of each element is a downstream offset;

[0024] The topic and the partition in the first offset mapping relationship corresponding to the consumer group are the same as the key of a target ordered set in the plurality of ordered sets;

[0025] The downstream offset in the first offset mapping relationship is the same as the value of a target element in the target ordered set;

[0026] The key of the target ordered set is the same as the topic and the partition in the second offset mapping relationship corresponding to the consumer group;

[0027] The score of the target element is the same as the upstream offset in the second offset mapping relationship.

[0028] In an implementation manner,

[0029] The first offset mapping relationship is further used for the database server to delete a target upstream-downstream corresponding relationship according to the second offset mapping relationship corresponding to the consumer group; wherein the topic and the partition in the target upstream-downstream corresponding relationship are the same as the topic and the partition in the second offset mapping relationship, and the downstream offset in the target upstream-downstream corresponding relationship is less than the downstream offset in the second offset mapping relationship.

[0030] In a second aspect, an embodiment of the present application provides a consumer offset information modification device, comprising:

[0031] The obtaining module is configured to obtain, for each message in the main message middleware cluster, a topic, a partition, an upstream offset, and a downstream offset corresponding to the message;

[0032] The processing module is configured to establish an upstream-downstream corresponding relationship according to the topic, the partition, the upstream offset, and the downstream offset corresponding to the message;

[0033] The transceiver module is configured to send the upstream-downstream corresponding relationship to the database server, so that the database server stores and processes the upstream-downstream corresponding relationship;

[0034] The processing module is further configured to determine a first offset mapping relationship corresponding to each consumer group according to first consumer offset information stored by the primary message middleware cluster; the first offset mapping relationship comprises a topic, a partition, and an upstream offset;

[0035] The transceiver module is further configured to send the first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine a second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships; the second offset mapping relationship comprises a topic, a partition, and a downstream offset;

[0036] The transceiver module is further configured to obtain the second offset mapping relationship corresponding to each consumer group sent by the database server.

[0037] The processing module is further configured to modify second consumer offset information stored by the backup message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0038] In an implementation manner,

[0039] The processing module is specifically configured to establish an upstream and downstream corresponding relationship according to a topic, a partition, an upstream offset, and a downstream offset corresponding to a message.

[0040] The processing module is specifically configured to store the upstream and downstream corresponding relationship to a local memory.

[0041] The transceiver module is specifically configured to send all upstream and downstream corresponding relationships stored in the local memory to the database server in a case where all upstream and downstream corresponding relationships stored in the local memory satisfy a preset condition.

[0042] In an implementation manner, the transceiver module is specifically configured to:

[0043] In a case where a number of all upstream and downstream corresponding relationships stored in the local memory is greater than a preset threshold, the transceiver module is configured to send all upstream and downstream corresponding relationships stored in the local memory to the database server.

[0044] In an implementation manner, the processing module is specifically configured to:

[0045] For each consumer group, the processing module is configured to determine a topic corresponding to the consumer group, and determine a partition included in the topic.

[0046] The processing module is configured to determine an upstream offset according to a consumer group, a partition, a topic, and first consumer offset information stored by the primary message middleware cluster, and determine a first offset mapping relationship according to the consumer group, the partition, the topic, and the upstream offset.

[0047] In an implementation manner,

[0048] The database server stores a plurality of ordered sets; a key of each ordered set is a topic and a partition; a score of each element in the ordered set is an upstream offset, and a value of each element is a downstream offset;

[0049] The topic and the partition in the first offset mapping relationship corresponding to the consumer group are the same as the key of the target ordered set in the plurality of ordered sets;

[0050] The downstream offset in the first offset mapping relationship is the same as the value of the target element in the target ordered set;

[0051] The key of the target ordered set is the same as the topic and the partition in the second offset mapping relationship corresponding to the consumer group;

[0052] The score of the target element is the same as the upstream offset in the second offset mapping relationship.

[0053] In an implementation manner,

[0054] The first offset mapping relationship is also used for the database server to delete a target upstream-downstream corresponding relationship according to the second offset mapping relationship corresponding to the consumer group; wherein the topic and the partition in the target upstream-downstream corresponding relationship are the same as the topic and the partition in the second offset mapping relationship, and the downstream offset in the target upstream-downstream corresponding relationship is less than the downstream offset in the second offset mapping relationship.

[0055] In a third aspect, an embodiment of the present application provides a computing device, comprising:

[0056] A processor, and a memory connected with the processor in communication;

[0057] The memory is used to store computer execution instructions;

[0058] The processor is used to execute the computer execution instructions stored in the memory, so as to implement the method of the first aspect.

[0059] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions, and the computer execution instructions are used to implement the method of the first aspect when executed by a processor.

[0060] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, and the computer program is used to implement the method of the first aspect when executed by a processor.

[0061] Embodiments of the present application provide a method, device, medium and product for modifying consumer offset information. For each message in a primary message middleware cluster, a computing device can obtain a topic, a partition, an upstream offset and a downstream offset corresponding to the message. The computing device can establish an upstream-downstream correspondence relationship according to the topic, the partition, the upstream offset and the downstream offset corresponding to the message, and store the upstream-downstream correspondence relationship in a database server. The computing device can determine a first offset mapping relationship corresponding to each consumer group according to first consumer offset information stored in the primary message middleware cluster; the first offset mapping relationship includes the topic, the partition and the upstream offset. The computing device can send the first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine a second offset mapping relationship corresponding to each consumer group according to the first offset mapping relationship and all upstream-downstream correspondence relationships; the second offset mapping relationship includes the topic, the partition and the downstream offset. The computing device can obtain the second offset mapping relationship corresponding to each consumer group sent by the database server, and modify second consumer offset information stored in a backup message middleware cluster according to the second offset mapping relationship corresponding to each consumer group. The method of the embodiments of the present application can establish the upstream-downstream correspondence relationship of each message, so that the computing device can accurately determine the second offset mapping relationship (including the downstream offset) corresponding to each consumer group, and then the computing device can accurately modify the second consumer offset information in the backup message middleware cluster, thereby improving the accuracy of the second consumer offset information in the backup message middleware cluster, and further improving the accuracy of the backup message middleware cluster in controlling the consumer to consume messages. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without any creative labor.

[0063] Figure 1 A schematic diagram of an application scenario applicable to the embodiments of the present application;

[0064] Figure 2a A flowchart of a first modification method for consumer offset information provided by the embodiments of the present application;

[0065] Figure 2b A schematic diagram of an ordered set provided by the embodiments of the present application;

[0066] Figure 3 Figure 2 is a flowchart illustrating a method for modifying consumer offset information according to an embodiment of the present application;

[0067] Figure 4 Figure 3 is a flowchart illustrating a method for modifying consumer offset information according to another embodiment of the present application;

[0068] Figure 5 Figure 4 is a schematic diagram of a device for modifying consumer offset information according to an embodiment of the present application;

[0069] Figure 6 Figure 5 is a schematic diagram of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments made by those of ordinary skill in the art under the inspiration of the embodiments of the present application belong to the scope of protection of the present application.

[0071] The terms "first", "second", "third", "fourth" and the like in the specification and claims of the present application and the above-described drawings (if any) are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0072] In the technical solutions of the present application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data comply with relevant laws and regulations and do not violate public order and good customs.

[0073] It should be noted that in the embodiments of the present application, some existing industry solutions such as software, components and models may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0074] Glossary:

[0075] Topic: A topic is a logical unit used in message middleware (such as Kafka) to classify and organize messages. Each topic can be seen as a message category or message stream. Producers can publish messages to topics, and consumers can subscribe to and consume messages from topics.

[0076] Partition: A partition is a physical shard of a topic. Each topic can be divided into at least one partition, and each partition is an ordered, immutable sequence of messages. Partitions enable Kafka to handle messages in parallel. Each partition can be processed by different consumers, thereby improving the throughput of the system. Through partitioning, the message middleware can distribute the load to multiple brokers. Within a single partition, messages are ordered.

[0077] The second consumer offset information stored by the standby message middleware needs to accurately reflect the progress of the consumer group consuming messages, so that when a cluster switch occurs (from the primary message middleware cluster to the standby message middleware cluster), the standby message middleware cluster can directly control the consumer to consume messages based on the second consumer offset information.

[0078] In related technologies, the computing device running the data migration tool can establish an upstream-downstream correspondence relationship based on the upstream offset of the message in the primary message middleware cluster and the downstream offset of the message in the standby message middleware cluster, and store the upstream-downstream correspondence relationship in the local memory. The computing device can synchronize and modify the second consumer offset information in the standby message middleware cluster based on the first consumer offset information in the primary message middleware cluster and the upstream-downstream correspondence relationship.

[0079] However, the method in related technologies establishes the upstream-downstream correspondence relationship of a message at an interval of at least one message, thereby causing the method in related technologies to have the problem that the computing device cannot accurately modify the second consumer offset information in the standby message middleware cluster, and further causing the method in related technologies to have the problem that the standby message middleware cluster controls the consumer to repeatedly consume messages.

[0080] To solve the above technical problems, the embodiment of the present application provides a method for modifying consumer offset information. In the method, for each message in the main message middleware cluster, the computing device can obtain the topic, partition, upstream offset and downstream offset corresponding to the message. The computing device can establish an upstream-downstream correspondence relationship according to the topic, partition, upstream offset and downstream offset corresponding to the message, and store the upstream-downstream correspondence relationship in the database server. The computing device can determine a first offset mapping relationship (including topic, partition and upstream offset) corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster. The computing device can send the first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine a second offset mapping relationship (including topic, partition and downstream offset) corresponding to the consumer group according to the first offset mapping relationship and all upstream-downstream correspondence relationships. The computing device can obtain the second offset mapping relationship corresponding to each consumer group sent by the database server, and modify the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0081] The method of the embodiment of the present application can establish the upstream-downstream correspondence relationship of each message, so that the computing device can accurately determine the second offset mapping relationship (including downstream offset) corresponding to each consumer group, and then the computing device can accurately modify the second consumer offset information in the standby message middleware cluster, thereby improving the accuracy of the second consumer offset information in the standby message middleware cluster, and further improving the accuracy of the standby message middleware cluster in controlling the consumer to consume messages.

[0082] To facilitate understanding, the application scenarios to which the embodiment of the present application is applicable will be described below in conjunction with an example of Figure 1 .

[0083] Exemplarily, Figure 1 a schematic diagram of an application scenario to which the embodiment of the present application is applicable. As shown in Figure 1 , the application scenario includes a main message middleware cluster 10, a computing device 20, a standby message middleware cluster 30 and a database server 40.

[0084] The computing device 20 is in communication connection with the main message middleware cluster 10, the standby message middleware cluster 30 and the database server 40 respectively.

[0085] It should be noted that the computing device 20 can be a terminal device (such as a computer), a server, or a server cluster. This is not limited in this embodiment of the present application. A data migration tool (such as MirrorMaker) can be run on the computing device 20. It should be noted that when MirrorMaker is run on the computing device 20, MirrorMaker includes three components: a first component (MirrorSourceConnector), a second component (MirrorCheckpointConnector), and a third component (MirrorHeartbeatConnector). MirrorMaker can perform data migration using these three components.

[0086] It should also be noted that message middleware can be run on the primary message middleware cluster 10 and the backup message middleware 30. For example, the message middleware can be Kafka, or other message middleware, which is not limited in the present embodiment.

[0087] The embodiment of this application is not Figure 1 The actual form of the equipment included in the Figure 1 The interaction mode between devices is limited and can be set according to actual needs in the application of the solution.

[0088] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0089] Figure 2a This is a flow chart of a first embodiment of a method for modifying consumer offset information provided by the present application. Figure 2a , the method specifically comprises the following steps:

[0090] S201: The computing device obtains the topic, partition, upstream offset, and downstream offset corresponding to each message in the main message middleware cluster.

[0091] In this embodiment, for each message in the primary message middleware cluster, the computing device may obtain the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0092] In one implementation, during data synchronization between the primary messaging middleware cluster and the backup messaging middleware cluster, the computing device may pull messages from the primary messaging middleware cluster and send the messages to the backup messaging middleware cluster.

[0093] The computing device can record the location of the message in a partition of a topic of the primary messaging middleware cluster and determine the location as the upstream offset. In addition, the computing device can record the topic and the partition in which the message is located.

[0094] The computing device can record the location of the message in a partition of a topic of the backup messaging middleware cluster and determine the location as the downstream offset. In addition, the computing device can record the topic and the partition in which the message is located. It should be noted that the topic and the partition in which the message is located in the primary messaging middleware cluster are the same as the topic and the partition in which the message is located in the backup messaging middleware cluster.

[0095] It should be noted that in an implementation, the computing device can save the topic, the partition, the upstream offset, and the downstream offset corresponding to the message in an offset synchronization information (offset_sync) of a topic of the primary messaging middleware cluster.

[0096] It should be noted that in the case that the data migration tool running on the computing device is MirrorMaker, a first component (MirrorSourceConnector) of the MirrorMaker can pull the message from the primary messaging middleware cluster and send the message to the backup messaging middleware cluster. The first component (MirrorSourceConnector) obtains the topic, the partition, the upstream offset, and the downstream offset corresponding to the message, and saves the topic, the partition, the upstream offset, and the downstream offset corresponding to the message in the offset synchronization information (offset_sync) of the primary messaging middleware cluster.

[0097] S202: The computing device establishes an upstream-downstream correspondence relationship according to the topic, the partition, the upstream offset, and the downstream offset corresponding to the message.

[0098] In the embodiment, the computing device can establish an upstream-downstream correspondence relationship according to the topic, the partition, the upstream offset, and the downstream offset corresponding to the message.

[0099] In an implementation, the computing device can pull the topic, the partition, the upstream offset, and the downstream offset corresponding to the message from the offset synchronization information (offset_sync) of the primary messaging middleware cluster. The computing device can establish an upstream-downstream correspondence relationship (offsetMap) according to the topic, the partition, the upstream offset, and the downstream offset corresponding to the message.

[0100] It should be noted that in an implementation, in the case that the data migration tool running on the computing device is MirrorMaker, the first component (MirrorSourceConnector) of the MirrorMaker can establish the upstream-downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0101] S203: The computing device sends the upstream-downstream correspondence relationship to the database server.

[0102] In this embodiment, the computing device can send the upstream-downstream correspondence relationship to the database server. It should be noted that in an implementation, the database server can be a server running a remote dictionary service (Remote Dictionary Server, Redis for short). It should be noted that from the perspective of software, it is the MirrorSourceConnector running on the computing device that calls the lua script (a lightweight and efficient script language) in Redis to send the upstream-downstream correspondence relationship to Redis.

[0103] In an implementation,

[0104] The computing device can store the upstream-downstream correspondence relationship in the local memory of the computing device.

[0105] The computing device can send all the upstream-downstream correspondence relationships stored in the local memory to the database server in the case that it is identified that all the upstream-downstream correspondence relationships stored in the local memory meet the preset condition. In an implementation, the computing device can send all the upstream-downstream correspondence relationships stored in the local memory to the database server in the case that it is identified that the number of all the upstream-downstream correspondence relationships stored in the local memory is greater than a preset threshold.

[0106] S204: The database server performs storage processing on the upstream-downstream correspondence relationship.

[0107] In this embodiment, the database server can perform storage processing on the upstream-downstream correspondence relationship after obtaining the upstream-downstream correspondence relationship sent by the computing device.

[0108] In an implementation, in the case that the database server is a server running Redis, the database can use a sorted set to store the upstream-downstream correspondence relationship.

[0109] The database server stores a plurality of sorted sets. Figure 2b A schematic diagram of a sorted set provided by an embodiment of the present application is shown. Exemplarily, Figure 2b Two sorted sets are shown.

[0110] The key of the ordered set is topic and partition. The ordered set includes multiple elements, wherein the score of the element is the upstream offset in the upstream-downstream correspondence, and the value of the element is the downstream offset in the upstream-downstream correspondence.

[0111] It should be noted that the database server can arrange the multiple elements in the ordered set in ascending order according to the score of each element in the ordered set.

[0112] S205: The computing device determines, according to the first consumer offset information stored by the main message middleware cluster, a first offset mapping relationship corresponding to each consumer group.

[0113] In this embodiment, the computing device can determine, according to the first consumer offset information stored by the main message middleware cluster, a first offset mapping relationship corresponding to each consumer group. The first offset mapping relationship includes topic, partition, and upstream offset.

[0114] It should be noted that the topic and the partition can be represented as “topicAndPartition”.

[0115] It should be further noted that each consumer group can subscribe to at least one topic, and each topic can include at least one partition, that is, each consumer group can consume messages in at least one partition of at least one topic. Therefore, each consumer group can correspond to at least one first offset mapping relationship.

[0116] In an implementation manner:

[0117] For each consumer group, the computing device can determine a topic corresponding to the consumer group, and determine a partition included in the topic. It should be noted that the topic corresponding to the consumer group refers to the topic subscribed by the consumer group. The topic can include the partition, and the partition is used to store messages. It should be further noted that the consumer group can correspond to at least one topic, and each topic can include at least one partition.

[0118] The computing device can determine, according to the consumer group, the partition, the topic, and the first consumer offset information stored in the main message cluster, an upstream offset. It should be noted that the first consumer offset information (_consumer_offsets) is stored in the main message middleware cluster. The first consumer offset information records the upstream offset corresponding to the relationship between the consumer group, the topic, and the partition. The upstream offset reflects the progress of the consumer in consuming information in the partition of the topic.

[0119] The computing device can determine the first offset mapping relationship according to the consumer group, the partition, the topic, and the upstream offset.

[0120] The computing device can store the first offset mapping relationship to the local memory after determining the first offset mapping relationship.

[0121] It should be noted that the first offset mapping relationship is stored in an offsetMap structure. It should also be noted that the topic and the partition in the first offset mapping relationship can be represented as "topicAndPartition".

[0122] It should also be noted that in an implementation, in the case where the data migration tool running on the computing device is MirrorMaker, the second component (MirrorCheckpointConnector) of the MirrorMaker determines the first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the main message middleware cluster.

[0123] S206: The computing device sends the first offset mapping relationship corresponding to each consumer group to the database server.

[0124] In this embodiment, the computing device can send the first offset mapping relationship corresponding to each consumer group to the database server after determining the first offset mapping relationship corresponding to all consumer groups.

[0125] It should be noted that based on the at least one first offset mapping relationship corresponding to the consumer group, the computing device can send the at least one first offset mapping relationship corresponding to all consumer groups to the database server after determining the at least one first offset mapping relationship corresponding to all consumer groups.

[0126] S207: The database server determines the second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and the complete upstream and downstream correspondence relationship.

[0127] In this embodiment, the database server can determine the second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and the complete upstream and downstream correspondence relationship.

[0128] In an implementation, in the case where the database server is a Redis server, the database server stores a plurality of ordered sets. The key of each ordered set is a topic and a partition (an identifier of the topic and an identifier of the partition); the score of each element in the ordered set is an upstream offset, and the value of each element is a downstream offset.

[0129] The first offset mapping relationship includes a topic, a partition (one partition included in the topic), and an upstream offset (the upstream offset reflects a consumption progress of the consumer group in the partition of the topic).

[0130] For each first offset mapping relationship, the database server can determine, from the plurality of ordered sets, an ordered set whose key is the same as the topic and the partition in the first offset mapping relationship, as a target ordered set. The database server can determine, from the target ordered set, an element whose score is the same as the upstream offset in the first offset mapping relationship, as a target element. The database server can determine a value of the target element as a downstream offset in the second offset mapping relationship. In one implementation, the database server can use a “zrangebyscore topic-partition offset offset” command to determine the downstream offset. It is noted that “topic-partition” refers to the topic and the partition, and the first offset and the second offset in “offset offset” are both the upstream offset in the first offset mapping relationship.

[0131] In addition, the database server can also determine, from the first offset mapping relationship, the topic and the partition as the topic and the partition in the second offset mapping relationship.

[0132] In this way, the second offset mapping relationship corresponding to the consumer group can be determined.

[0133] It is noted that the topic and the partition in the first offset mapping relationship corresponding to the consumer group are the same as the key of the target ordered set from the plurality of ordered sets. The downstream offset in the first offset mapping relationship is the same as the value of the target element in the target ordered set. The key of the target ordered set is the same as the topic and the partition in the second offset mapping relationship corresponding to the consumer group. The score of the target element is the same as the upstream offset in the second offset mapping relationship.

[0134] It can be understood that the topic and the partition in the second offset mapping relationship corresponding to a consumer group are the same as the topic and the partition in the first offset mapping relationship corresponding to the consumer group. The downstream offset in the second offset mapping relationship corresponding to the consumer group is the downstream offset of a message, and the upstream offset in the first offset mapping relationship corresponding to the consumer group is the upstream offset of the message.

[0135] That is, the second offset mapping relationship corresponding to a consumer group corresponds to the first offset mapping relationship corresponding to the consumer group. It should be noted that, in the case where one consumer group corresponds to N (N is a positive integer greater than or equal to 1) first offset mapping relationships, the consumer group corresponds to N second offset mapping relationships, and each first offset mapping relationship corresponds to a second offset mapping relationship.

[0136] In addition, it should be noted that the second offset mapping relationship is stored in the resultMap structure.

[0137] S208: The computing device obtains the second offset mapping relationship corresponding to each consumer group sent by the database server.

[0138] In the embodiment, the computing device can obtain the second offset mapping relationship corresponding to each consumer group sent by the database server. In an implementation, in the case where each consumer group corresponds to at least one second offset mapping relationship, the computing device can obtain at least one second offset mapping relationship corresponding to each consumer group sent by the database server.

[0139] It can be understood that, in an implementation, the computing device can obtain the resultMap structure (storing the second offset mapping relationship corresponding to each consumer group) sent by the database server.

[0140] S209: The computing device modifies the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0141] In the embodiment, the standby message middleware cluster stores the second consumer offset information (_consumer_offset). The second consumer offset information includes the topic, partition, and downstream offset corresponding to the consumer group.

[0142] The computing device can modify the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group, so that when the switching between the primary message middleware cluster and the standby message middleware cluster occurs, the standby message middleware cluster can directly control the consumer (Consumer) to consume messages based on the second consumer offset information, avoiding the problem of repeated consumption, and improving the accuracy of consuming messages.

[0143] The beneficial effects of the embodiment are: for each message in the main message middleware cluster, the computing device can obtain the topic, partition, upstream offset, and downstream offset corresponding to the message. The computing device can establish an upstream-downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message, and store the upstream-downstream correspondence relationship to the database server. The computing device can determine a first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored by the main message middleware cluster; the first offset mapping relationship includes the topic, partition, and upstream offset. The computing device can send the first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine a second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream-downstream correspondence relationships; the second offset mapping relationship includes the topic, partition, and downstream offset. The computing device can obtain the second offset mapping relationship corresponding to each consumer group sent by the database server, and modify the second consumer offset information stored by the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group. The method of the embodiment of the application can establish the upstream-downstream correspondence relationship of each message, so that the computing device can accurately determine the second offset mapping relationship (including the downstream offset) corresponding to each consumer group, and then the computing device can accurately modify the second consumer offset information in the standby message middleware cluster, thereby improving the accuracy of the second consumer offset information in the standby message middleware cluster, and further improving the accuracy of the standby message middleware cluster in controlling the consumer to consume the message.

[0144] Figure 3 An embodiment of a flowchart of a method for modifying consumer offset information provided by the embodiment of the application is provided. Referring to FIG. 8, the method specifically includes the following steps: Figure 3

[0145] S301: The computing device obtains the topic, partition, upstream offset, and downstream offset corresponding to each message in the main message middleware cluster.

[0146] In the embodiment, for each message in the main message middleware cluster, the computing device can obtain the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0147] The specific implementation process is the same as that of S201, which will not be described here.

[0148] S302: The computing device establishes an upstream-downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message.

[0149] ​In the embodiment, the computing device can establish the upstream-downstream correspondence relationship according to the topic, partition, upstream offset and downstream offset corresponding to the message.

[0150] The specific implementation process is the same as that of S202, which will not be described here.

[0151] S303: The computing device stores the upstream-downstream correspondence relationship in the local memory.

[0152] In the embodiment, the computing device can store the upstream-downstream correspondence relationship corresponding to each message in the local memory after establishing the upstream-downstream correspondence relationship.

[0153] S304: The computing device sends all the upstream-downstream correspondence relationships stored in the local memory to the database server in a case where all the upstream-downstream correspondence relationships stored in the local memory meet a preset condition.

[0154] In the embodiment, the computing device can send all the upstream-downstream correspondence relationships stored in the local memory to the database server in a case where all the upstream-downstream correspondence relationships stored in the local memory meet a preset condition.

[0155] In an implementation manner, the computing device can send all the upstream-downstream correspondence relationships stored in the local memory to the database server in a case where the number of all the upstream-downstream correspondence relationships stored in the local memory is greater than a preset threshold. Exemplarily, the preset threshold can be 1000.

[0156] In an implementation manner, the computing device can send all the upstream-downstream correspondence relationships stored in the local memory to the database server in a case where the number of all the upstream-downstream correspondence relationships stored in the local memory is greater than a preset threshold and the memory occupancy rate of the local memory is greater than a preset occupancy rate.

[0157] S305: The database server performs storage processing on the upstream-downstream correspondence relationship.

[0158] In the embodiment, the database server can perform storage processing on the upstream-downstream correspondence relationship after obtaining the upstream-downstream correspondence relationship sent by the computing device.

[0159] The specific implementation process is the same as that of S204, which will not be described here.

[0160] S306: The computing device determines a first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the master message middleware cluster.

[0161] In the embodiment, the computing device can determine the first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored by the primary message middleware cluster. The first offset mapping relationship includes a topic, a partition, and an upstream offset.

[0162] The specific implementation process is the same as that of S205, and details are not repeated here.

[0163] S307: The computing device sends the first offset mapping relationship corresponding to each consumer group to the database server.

[0164] In the embodiment, the computing device can send the first offset mapping relationship corresponding to each consumer group to the database server.

[0165] The specific implementation process is the same as that of S206, and details are not repeated here.

[0166] S308: The database server determines the second offset mapping relationship corresponding to each consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships.

[0167] In the embodiment, the database server can determine the second offset mapping relationship corresponding to each consumer group according to the first offset mapping relationship and all upstream and downstream corresponding relationships.

[0168] The specific implementation process is the same as that of S207, and details are not repeated here.

[0169] S309: The computing device obtains the second offset mapping relationship corresponding to each consumer group sent by the database server.

[0170] In the embodiment, the computing device can obtain the second offset mapping relationship corresponding to each consumer group sent by the database server.

[0171] The specific implementation process is the same as that of S208, and details are not repeated here.

[0172] S310: The computing device modifies the second consumer offset information stored by the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0173] In the embodiment, the computing device can modify the second consumer offset information stored by the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0174] The specific implementation process is the same as that of S209, and details are not repeated here.

[0175] Advantages of the present embodiment: The computing device can establish an upstream-downstream correspondence relationship according to the upstream offset, the downstream offset, the topic, and the partition corresponding to the message, and store the upstream-downstream correspondence relationship to the local memory. The computing device can send all the upstream-downstream correspondence relationships stored in the local memory to the database server in the case that all the upstream-downstream correspondence relationships stored in the local memory meet the preset condition. In this way, the number of interactions between the computing device and the database server can be reduced, the communication overhead of the computing device and the database server is saved, and the storage efficiency of the upstream-downstream correspondence relationship is improved.

[0176] Figure 4 A flowchart of a consumer offset information modification method embodiment four provided by the present application is shown in FIG. 4. As shown in FIG. 4, the method comprises the following steps: Figure 4

[0177] S401: The computing device obtains the topic, the partition, the upstream offset, and the downstream offset corresponding to each message in the main message middleware cluster.

[0178] In the present embodiment, the computing device can obtain the topic, the partition, the upstream offset, and the downstream offset corresponding to each message in the main message middleware cluster.

[0179] The specific implementation process is the same as that of S201, which will not be described here.

[0180] S402: The computing device establishes an upstream-downstream correspondence relationship according to the topic, the partition, the upstream offset, and the downstream offset corresponding to the message.

[0181] In the present embodiment, the computing device can establish an upstream-downstream correspondence relationship according to the topic, the partition, the upstream offset, and the downstream offset corresponding to the message.

[0182] The specific implementation process is the same as that of S202, which will not be described here.

[0183] S403: The computing device stores the upstream-downstream correspondence relationship to the local memory.

[0184] In the present embodiment, the computing device can store the upstream-downstream correspondence relationship corresponding to each message to the local memory after establishing the upstream-downstream correspondence relationship.

[0185] S404: The computing device sends all the upstream-downstream correspondence relationships stored in the local memory to the database server in the case that all the upstream-downstream correspondence relationships stored in the local memory meet the preset condition.

[0186] ​In the embodiment, the computing device can send all the upstream-downstream correspondence relationships stored in the local memory to the database server when it is identified that all the upstream-downstream correspondence relationships stored in the local memory meet the preset condition.

[0187] The specific implementation process is the same as that of S304, which will not be described here.

[0188] S405: The database server stores the upstream-downstream correspondence relationship.

[0189] In the embodiment, the database server can store the upstream-downstream correspondence relationship after obtaining the upstream-downstream correspondence relationship sent by the computing device.

[0190] The specific implementation process is the same as that of S204, which will not be described here.

[0191] S406: The computing device determines a first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored by the master message middleware cluster.

[0192] In the embodiment, the computing device can determine a first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored by the master message middleware cluster. The first offset mapping relationship includes a topic, a partition, and an upstream offset.

[0193] The specific implementation process is the same as that of S205, which will not be described here.

[0194] S407: The computing device sends the first offset mapping relationship corresponding to each consumer group to the database server.

[0195] In the embodiment, the computing device can send the first offset mapping relationship corresponding to each consumer group to the database server.

[0196] The specific implementation process is the same as that of S206, which will not be described here.

[0197] S408: The database server determines a second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all the upstream-downstream correspondence relationships.

[0198] In the embodiment, the database server can determine a second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all the upstream-downstream correspondence relationships.

[0199] The specific implementation process is the same as that of S207, which will not be described here.

[0200] S409: The database server deletes the target upstream-downstream correspondence relationship according to the second offset mapping relationship corresponding to the consumer group.

[0201] In the embodiment, after determining the second offset mapping relationship corresponding to the consumer group, the database server can delete the target upstream-downstream corresponding relationship according to the second offset mapping relationship.

[0202] The topic and the partition in the target upstream-downstream corresponding relationship are the same as those in the second offset mapping relationship, and the downstream offset in the target upstream-downstream corresponding relationship is less than the downstream offset in the second offset mapping relationship.

[0203] In an implementation manner, the database server can use the “zremrangebyscore 0 offset” command to delete the target upstream-downstream corresponding relationship. It should be noted that the “offset” refers to the downstream offset in the second offset mapping relationship.

[0204] It should be noted that in an implementation manner, after determining the second offset mapping relationship corresponding to one consumer group, the database server can delete at least one target upstream-downstream corresponding relationship according to the second offset mapping relationship. In an implementation manner, after determining the second offset mapping relationship corresponding to all consumer groups, the database server can delete all target upstream-downstream corresponding relationships according to the second offset mapping relationship corresponding to all consumers.

[0205] S410: The computing device obtains the second offset mapping relationship corresponding to each consumer group sent by the database server.

[0206] In the embodiment, the computing device can obtain the second offset mapping relationship corresponding to each consumer group sent by the database server.

[0207] The specific implementation process is the same as that of S208, which will not be described here.

[0208] S411: The computing device modifies the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0209] In the embodiment, the computing device can modify the second consumer offset information stored in the standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0210] The specific implementation process is the same as that of S209, which will not be described here.

[0211] Advantages of the present embodiment: in the present embodiment, after the database server determines a second offset mapping relationship corresponding to a consumer group, the database server can delete a target upstream-downstream correspondence relationship according to the second offset mapping relationship. The topic and partition in the target upstream-downstream correspondence relationship are the same as those in the second offset mapping relationship, and the downstream offset in the target upstream-downstream correspondence relationship is less than the downstream offset in the second offset mapping relationship. Through the above manner, the upstream-downstream correspondence relationship corresponding to the consumed messages in the database server can be deleted in time, avoiding the waste of space of the database server to store the target upstream-downstream correspondence relationship, and further causing the storage space of the database server to be occupied. That is, through the above manner, the utilization rate of the memory space of the database server is improved.

[0212] Figure 5 A structure schematic diagram of a consumer offset information modification device provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the consumer offset information modification device 50 includes an acquisition module 51, a processing module 52, and a transceiver module 53. Among them, Figure 5

[0213] The acquisition module 51 is configured to acquire, for each message in the master message middleware cluster, a topic, a partition, an upstream offset, and a downstream offset corresponding to the message.

[0214] The processing module 52 is configured to establish an upstream-downstream correspondence relationship according to the topic, the partition, the upstream offset, and the downstream offset corresponding to the message.

[0215] The transceiver module 53 is configured to send the upstream-downstream correspondence relationship to a database server, so that the database server stores and processes the upstream-downstream correspondence relationship.

[0216] The processing module 52 is further configured to determine a first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored by the master message middleware cluster. The first offset mapping relationship includes a topic, a partition, and an upstream offset.

[0217] The transceiver module 53 is further configured to send the first offset mapping relationship corresponding to each consumer group to the database server. The first offset mapping relationship is used for the database server to determine a second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all the upstream-downstream correspondence relationships. The second offset mapping relationship includes a topic, a partition, and a downstream offset.

[0218] The transceiver module 53 is further configured to acquire the second offset mapping relationship corresponding to each consumer group sent by the database server.

[0219] ​The processing module 52 is further configured to modify the second consumer offset information stored in the backup message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

[0220] The modification device for consumer offset information provided in the embodiments of the present application can execute the technical solutions shown in the method embodiments, and has similar implementation principles and beneficial effects, which will not be repeated here.

[0221] In an implementation manner,

[0222] The processing module 52 is specifically configured to establish the upstream-downstream correspondence relationship according to the upstream offset, the downstream offset, the topic, and the partition corresponding to the message.

[0223] The processing module 52 is specifically configured to store the upstream-downstream correspondence relationship in the local memory.

[0224] The transceiver module 53 is specifically configured to send all the upstream-downstream correspondence relationships stored in the local memory to the database server in a case where it is identified that all the upstream-downstream correspondence relationships stored in the local memory meet the preset condition.

[0225] The modification device for consumer offset information provided in the embodiments of the present application can execute the technical solutions shown in the method embodiments, and has similar implementation principles and beneficial effects, which will not be repeated here.

[0226] In an implementation manner, the transceiver module 53 is specifically configured to:

[0227] In a case where the number of all the upstream-downstream correspondence relationships stored in the local memory is greater than the preset threshold, the transceiver module 53 is specifically configured to send all the upstream-downstream correspondence relationships stored in the local memory to the database server.

[0228] The modification device for consumer offset information provided in the embodiments of the present application can execute the technical solutions shown in the method embodiments, and has similar implementation principles and beneficial effects, which will not be repeated here.

[0229] In an implementation manner, the processing module 52 is specifically configured to:

[0230] For each consumer group, the processing module 52 is specifically configured to determine a topic corresponding to the consumer group, and determine a partition included in the topic.

[0231] The processing module 52 is specifically configured to determine the upstream offset according to the consumer group, the partition, the topic, and the first consumer offset information stored in the main message middleware cluster, and determine the first offset mapping relationship according to the consumer group, the partition, the topic, and the upstream offset.

[0232] The consumer offset information modification apparatus provided in the embodiments of the present application can execute the technical solutions shown in the method embodiments, and the implementation principles and beneficial effects are similar, which will not be repeated here.

[0233] In an implementation manner,

[0234] The database server stores a plurality of ordered sets; the key of each ordered set is a topic and a partition; the score of each element in the ordered set is an upstream offset, and the value of each element is a downstream offset;

[0235] The topic and the partition in the first offset mapping relationship corresponding to the consumer group are the same as the key of the target ordered set in the plurality of ordered sets;

[0236] The downstream offset in the first offset mapping relationship is the same as the value of the target element in the target ordered set;

[0237] The key of the target ordered set is the same as the topic and the partition in the second offset mapping relationship corresponding to the consumer group;

[0238] The score of the target element is the same as the upstream offset in the second offset mapping relationship.

[0239] The consumer offset information modification apparatus provided in the embodiments of the present application can execute the technical solutions shown in the method embodiments, and the implementation principles and beneficial effects are similar, which will not be repeated here.

[0240] In an implementation manner,

[0241] The first offset mapping relationship is also used for the database server to delete a target upstream-downstream corresponding relationship according to the second offset mapping relationship corresponding to the consumer group; wherein the topic and the partition in the target upstream-downstream corresponding relationship are the same as the topic and the partition in the second offset mapping relationship, and the downstream offset in the target upstream-downstream corresponding relationship is less than the downstream offset in the second offset mapping relationship.

[0242] The consumer offset information modification apparatus provided in the embodiments of the present application can execute the technical solutions shown in the method embodiments, and the implementation principles and beneficial effects are similar, which will not be repeated here.

[0243] Figure 6 A structural diagram of a computing device is provided in the present application. As shown in the figure, Figure 6 The computing device 60 includes a processor 61 and a memory 62. The processor 61 is in communication connection with the memory 62, the memory 62 is used to store computer execution instructions; the processor 61 is configured to execute the technical solutions in any of the preceding method embodiments by executing the computer execution instructions stored in the memory 62.

[0244] Optionally, the memory 62 can be independent or integrated with the processor 61. Optionally, when the memory 62 is independent of the processor 61, the computing device 60 can further include a bus 63 for connecting the above-mentioned devices.

[0245] The computing device is configured to execute the technical solutions in any of the preceding method embodiments, and has similar implementation principles and technical effects, which will not be repeated here.

[0246] The embodiments of the present application further provide a computer readable storage medium, and the computer readable storage medium stores computer execution instructions. When the computer execution instructions are executed by a processor, the computer execution instructions are configured to implement the technical solutions provided by any of the preceding method embodiments.

[0247] The embodiments of the present application further provide a computer program product, and the computer program product includes a computer program. When the computer program is executed by a processor, the computer program is configured to implement the technical solutions provided by the preceding method embodiments.

[0248] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by a program instruction related hardware. The above-mentioned program can be stored in a computer readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments; and the above-mentioned storage medium includes a ROM, a RAM, a magnetic disk or an optical disk and various storage media that can store program codes.

[0249] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the above-mentioned embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of modifying consumer offset amount information, characterized by, include: For each message in the main message middleware cluster, obtain the corresponding topic, partition, upstream offset, and downstream offset of the message; Establish an upstream and downstream correspondence relationship according to the topic, partition, upstream offset, and downstream offset corresponding to the message, and send the upstream and downstream correspondence relationship to the database server so that the database server stores and processes the upstream and downstream correspondence relationship; Determine a first offset mapping relationship corresponding to each consumer group according to the first consumer offset information stored in the primary message middleware cluster; the first offset mapping relationship includes a topic, a partition, and an upstream offset; Sending the first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine the second offset mapping relationship corresponding to the consumer group based on the first offset mapping relationship and all upstream and downstream correspondences; the second offset mapping relationship includes a topic, a partition, and a downstream offset; Obtaining a second offset mapping relationship corresponding to each consumer group sent by the database server; According to the second offset mapping relationship corresponding to each consumer group, the second consumer offset information stored in the standby message middleware cluster is modified.

2. The method of claim 1, wherein, The establishing an upstream and downstream correspondence relationship according to the upstream offset, downstream offset, topic, and partition corresponding to the message, and sending the upstream and downstream correspondence relationship to the database server includes: Establishing the upstream and downstream correspondence relationship according to the upstream offset, downstream offset, topic, and partition corresponding to the message; Storing the upstream and downstream correspondence in local memory; When all upstream and downstream correspondences stored in the local memory are identified and a preset condition is met, all upstream and downstream correspondences stored in the local memory are sent to the database server.

3. The method of claim 2, wherein, The step of sending all upstream and downstream corresponding relationships stored in the local memory to the database server upon identifying all upstream and downstream corresponding relationships stored in the local memory and satisfying a preset condition includes: When it is identified that the number of all upstream and downstream correspondences stored in the local memory is greater than a preset threshold, all upstream and downstream correspondences stored in the local memory are sent to the database server.

4. The method of claim 1, wherein, The determining, according to the first consumer offset information stored in the primary message middleware cluster, a first offset mapping relationship corresponding to each consumer group includes: For each consumer group, determine the topic corresponding to the consumer group and the partitions included in the topic; Determine an upstream offset according to the consumer group, the partition, the topic, and the first consumer offset information stored in the primary message middleware cluster, and determine a first offset mapping relationship according to the consumer group, the partition, the topic, and the upstream offset.

5. The method according to claim 4, characterized in that The database server stores a plurality of ordered sets; the key of each ordered set is a topic and a partition; the score of each element in the ordered set is an upstream offset, and the value of each element is a downstream offset; The topic and the partition in the first offset mapping relationship corresponding to the consumer group are the same as the key of a target ordered set in the plurality of ordered sets; The downstream offset in the first offset mapping relationship is the same as the value of a target element in the target ordered set; The key of the target ordered set is the same as the topic and the partition in the second offset mapping relationship corresponding to the consumer group; The score of the target element is the same as the upstream offset in the second offset mapping relationship.

6. The method of any one of claims 1-5, wherein The first offset mapping relationship is further used for the database server to delete a target upstream-downstream corresponding relationship according to the second offset mapping relationship corresponding to the consumer group; wherein the topic and the partition in the target upstream-downstream corresponding relationship are the same as the topic and the partition in the second offset mapping relationship, and the downstream offset in the target upstream-downstream corresponding relationship is less than the downstream offset in the second offset mapping relationship.

7. A consumer offset amount information modification apparatus characterized by comprising: Comprising: An acquisition module, configured to acquire, for each message in a master message middleware cluster, a topic, a partition, an upstream offset, and a downstream offset corresponding to the message; A processing module, configured to establish an upstream-downstream corresponding relationship according to the topic, the partition, the upstream offset, and the downstream offset corresponding to the message; A transceiver module, configured to send the upstream-downstream corresponding relationship to a database server, so that the database server stores and processes the upstream-downstream corresponding relationship; The processing module is further configured to determine a first offset mapping relationship corresponding to each consumer group according to first consumer offset information stored by the master message middleware cluster; the first offset mapping relationship includes a topic, a partition, and an upstream offset; The transceiver module is further configured to send the first offset mapping relationship corresponding to each consumer group to the database server; the first offset mapping relationship is used for the database server to determine a second offset mapping relationship corresponding to the consumer group according to the first offset mapping relationship and all upstream-downstream corresponding relationships; the second offset mapping relationship includes a topic, a partition, and a downstream offset; The transceiver module is further configured to acquire the second offset mapping relationship corresponding to each consumer group sent by the database server; The processing module is further configured to modify second consumer offset information stored by a standby message middleware cluster according to the second offset mapping relationship corresponding to each consumer group.

8. A computing device, comprising: Comprising: A processor, and a memory in communication connection with the processor; The memory is used to store computer execution instructions; The processor is used to execute the computer execution instructions stored in the memory, so as to realize the method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterised in that, The invention comprises a computer program, which is used to implement the method according to any one of claims 1 to 6 when the computer program is executed by a processor.

Images