Method, device, equipment and storage medium for data processing in message queue

Abstract

The present application discloses a method, device, equipment and storage medium for processing data in a message queue. The method includes: obtaining the offset information of the data in the message queue of the target subject and the consumption progress of the data consumer for the data in the message queue of the target subject, the offset information of the data includes the offset of the data in the message queue within at least one statistical period, and the consumption progress includes the offset of the data consumed by the data consumer in the message queue; determining the lag time of the data consumer according to the offset information and the consumption progress; adjusting the data storage time of the message queue according to the lag time. This method reduces the risk of data loss.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for processing data in a message queue. Background Art

[0002] With the development of information technology, there is a need for data sharing between application systems within enterprises. Kafka is a widely used distributed message queue system that can meet the need for data sharing. Specifically, Kafka message subscribers (data consumers) usually consume data produced by message producers (data providers) from a partition in a topic.

[0003] In some scenarios, the speed at which data consumers consume data is lower than the speed at which data providers produce data, which can cause a backlog of data in the partition. When the data in a partition exceeds the preset storage time, Kafka's cleanup mechanism is triggered, resulting in data loss.

[0004] Therefore, when the data consumer consumes data at a speed lower than the data provider produces data, there is a risk of data loss. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a method, device, equipment and storage medium for data processing in a message queue, which can avoid the problem of data loss caused by the cleaning mechanism of Kafka.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] In a first aspect, the present application provides a method for processing data in a message queue, comprising:

[0008] Obtain the offset information of the data in the message queue of the target topic and the consumption progress of the data in the message queue of the target topic by the data consumer, wherein the offset information of the data includes the offset of the data in the message queue within at least one statistical period, and the consumption progress includes the offset of the data consumed by the data consumer in the message queue;

[0009] Determine the lag time of the data consumer according to the offset information and the consumption progress;

[0010] According to the lag time, the data storage time of the message queue is adjusted.

[0011] Optionally, the method further includes: when the lag time is greater than or equal to an alarm time threshold, feeding back alarm information to the data consumer.

[0012] Optionally, the method further includes:

[0013] Determine the amount of lagged data of the data consumer according to the offset information and the consumption progress;

[0014] When the amount of delayed data is greater than or equal to the alarm data amount threshold, alarm information is fed back to the data consumer.

[0015] Optionally, the method further includes:

[0016] Obtain the alarm configuration information of the data consumer;

[0017] An alarm condition is generated according to the alarm configuration information, and the alarm condition is used to trigger an alarm process when it is met.

[0018] Optionally, adjusting the data storage time of the message queue according to the lag time includes:

[0019] When the lag time is greater than or equal to the lag time threshold, adjusting the data storage time of the message queue;

[0020] or,

[0021] When the difference between the lag time and the data retention time of the message queue is lower than or equal to the remaining time threshold, the data retention time of the message queue is adjusted.

[0022] Optionally, adjusting the data storage time of the message queue includes:

[0023] If the sum of the lag time and the data retention time of the message queue is greater than or equal to the maximum data retention time of the message queue, adjust the data retention time to the maximum data retention time;

[0024] If the sum of the lag time and the data retention time of the message queue is less than the maximum data retention time of the message queue, adjust the data retention time to the sum of the data retention time and the lag time.

[0025] In a second aspect, the present application provides a device for processing data in a message queue, including: an acquisition module, a determination module, and an adjustment module;

[0026] The acquisition module is used to acquire the offset information of the data in the message queue of the target topic and the consumption progress of the data consumer for the data in the message queue of the target topic, wherein the data offset information includes the offset of the data in the message queue within at least one statistical period, and the consumption progress includes the offset of the data consumed by the data consumer in the message queue;

[0027] The determination module is used to determine the lag time of the data consumer according to the offset information and the consumption progress;

[0028] The adjustment module is used to adjust the data storage time of the message queue according to the lag time.

[0029] Optionally, the device further includes: an alarm module;

[0030] The alarm module is used to feed back alarm information to the data consumer when the lag time is greater than or equal to the alarm time threshold.

[0031] Optionally, the device further includes: an alarm module;

[0032] The determination module is further used to determine the amount of lagged data of the data consumer according to the offset information and the consumption progress;

[0033] The alarm module is used to feed back alarm information to the data consumer when the amount of delayed data is greater than or equal to an alarm data amount threshold.

[0034] Optionally, the acquisition module is further used to acquire alarm configuration information of the data consumer;

[0035] An alarm condition is generated according to the alarm configuration information, and the alarm condition is used to trigger an alarm process when it is met.

[0036] Optionally, the adjustment module is specifically used to adjust the data storage time of the message queue when the lag time is greater than or equal to the lag time threshold;

[0037] or,

[0038] When the difference between the lag time and the data retention time of the message queue is lower than or equal to the remaining time threshold, the data retention time of the message queue is adjusted.

[0039] Optionally, the adjustment module is specifically configured to adjust the data retention time to the maximum data retention time if the sum of the lag time and the data retention time of the message queue is greater than or equal to the maximum data retention time of the message queue;

[0040] If the sum of the lag time and the data retention time of the message queue is less than the maximum data retention time of the message queue, adjust the data retention time to the sum of the data retention time and the lag time.

[0041] In a third aspect, the present application provides a data processing device in a message queue, the device comprising: a memory and a processor;

[0042] The memory is used to store a computer program and transmit the computer program to the processor;

[0043] The processor executes the method for processing data in a message queue according to any one of the first aspects described above according to the instructions in the computer program.

[0044] In a third aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium is used to store computer software instructions used by a data processing system in a message queue, and when the computer-readable storage medium is run on a computer, the computer can execute the data processing method in a message queue described in any one of the first aspects above.

[0045] It can be seen from the above technical solution that the present application has the following beneficial effects:

[0046] The method obtains the offset information of the data in the message queue of the target topic and the consumption progress of the data consumer for the data in the message queue of the target topic. Since the offset information includes the offset of the data in the message queue within at least one statistical period, and the message progress includes the offset of the data already consumed by the data consumer in the message queue, the method can determine the lag time of the data consumer based on the offset information and the consumption progress, that is, the lag time is the lag time between the data in the message queue being consumed by the data consumer and the latest data in the message queue, and then adjust the data storage time of the message queue based on the lag time, thereby avoiding the situation where the data in the message queue is cleared due to exceeding the data storage time. Therefore, the method can reduce the risk of data loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A system architecture diagram of a monitoring system provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of offset information provided in an embodiment of the present application;

[0050] Figure 3 A flowchart of a method for processing data in a message queue provided in an embodiment of the present application;

[0051] Figure 4A flowchart of a method for processing data in a message queue provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a device for processing data in a message queue provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of a device for processing data in a message queue provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] To facilitate understanding, the technical terms involved in this application are first introduced below.

[0056] A message queue is a container that stores messages during their transmission. In the distributed message queue system Kafka, data providers push data to Kafka topics so that data consumers can consume the data in the topics. Generally, data consumers consume data in the order in which data providers write data to the topics.

[0057] In some scenarios, the data consumer consumes data at a slower rate than the data provider writes data, which results in a backlog of data in the topic. In order to ensure that data written by the data provider can be received, Kafka triggers a cleanup mechanism when the data in the topic exceeds the preset storage time. As a result, some data is cleared before it is consumed by the data consumer, resulting in data loss, which in turn affects the business.

[0058] In view of this, the present application provides a method for processing data in a message queue. The data processing method can be implemented by a monitoring system. Specifically, the monitoring system obtains the offset information of the data in the message queue of the target topic and the consumption progress of the data consumer for the data in the message queue of the target topic. The offset information of the data includes the offset of the data in the message queue within at least one statistical period, and the message progress includes the offset of the data consumed by the data consumer in the message queue. Then, the lag time of the data consumer is determined according to the offset information and the consumption progress, and then the data storage time in the message queue is adjusted according to the lag time.

[0059] The method obtains the offset information of the data in the message queue of the target topic and the consumption progress of the data consumer for the data in the message queue of the target topic. Since the offset information includes the offset of the data in the message queue within at least one statistical period, and the message progress includes the offset of the data already consumed by the data consumer in the message queue, the method can determine the lag time of the data consumer based on the offset information and the consumption progress, that is, the lag time is the lag time between the data in the message queue being consumed by the data consumer and the latest data in the message queue, and then adjust the data storage time of the message queue based on the lag time, thereby avoiding the situation where the data in the message queue is cleared due to exceeding the data storage time. Therefore, the method can reduce the risk of data loss.

[0060] The monitoring system provided in the embodiment of the present application can be used for data sharing, for example, data sharing between application systems within an enterprise. In some scenarios, application system A can push data to a topic in Kafka as a data provider, and other application systems (for example, application system B that has subscribed to the topic) can obtain data in the topic as data consumers, thereby realizing data sharing between application system A and other application systems.

[0061] The monitoring system may be a software system. Specifically, the monitoring system may be deployed in a computing device in the form of computer software to implement the function of adjusting the data retention time of the message queue. In some embodiments, the monitoring system may also be a hardware system. The hardware system includes a physical device having the function of adjusting the data retention time in the message queue.

[0062] The monitoring system is implemented by subsystems with different functions and units with different functions. The embodiment of the present application does not limit the division of subsystems and units within the monitoring system. Figure 1 An exemplary division method is shown in FIG.

[0063] like Figure 1 As shown, the monitoring system 100 includes a communication unit 110, a hysteresis unit 120, an adjustment unit 130, an alarm unit 140 and a condition unit 150. The alarm unit 140 and the condition unit 150 are optional and may not be required in some embodiments.

[0064] The communication unit 110 is used to obtain the offset information of the data in the message queue of the target topic, and the offset information includes the offset of the data in the message queue within at least one statistical period, see Figure 2 , which shows a schematic diagram of offset information, from Figure 2It can be seen that the offset information includes the offsets of multiple statistical periods, for example: the offset corresponding to 1:25 is 1000, the offset corresponding to 1:30 is 2000, and the offset corresponding to 4:00 is 12600. The communication unit 110 is also used to obtain the consumption progress of the data in the message queue of the target topic by the data consumer, and the consumption progress includes the offset of the data consumed by the data consumer in the message queue. For example, the consumption progress can be the offset of the data currently consumed by the data consumer.

[0065] The hysteresis unit 120 is used to determine the hysteresis time of the data consumer according to the offset information and consumption progress obtained by the communication unit 110. The hysteresis time refers to the time difference between the statistical moment of the data being consumed by the data consumer and the current moment. Figure 2 Taking the offset information shown in as an example, if the consumption progress indicates that the offset of the data consumed by the data consumer is 2000, then it can be determined that the statistical time of the data is 1:30. If the current time is 4:00, it can be determined that the lag time is 150 minutes.

[0066] The adjustment unit 130 is used to adjust the data retention time of the message queue according to the lag time. Due to the cleaning mechanism of Kafka itself, when the lag time of the data in the message queue exceeds the data retention time, the data in the message queue will be cleared. In this application, in order to avoid the data being cleared, before the lag time exceeds the data retention time, the data retention time is adjusted, for example, the data retention time is delayed, thereby avoiding the situation where the data in the message queue is cleared. Thus, the monitoring system 100 can reduce the risk of data loss.

[0067] In some implementations, the adjustment unit 130 can determine the relationship between the hysteresis time and the hysteresis time threshold, wherein the hysteresis time threshold can be preset and is less than the data retention time. When the hysteresis time is greater than or equal to the hysteresis time threshold, the data retention time of the message queue is adjusted.

[0068] In other implementations, the adjustment unit 130 can determine the relationship between the lag time and the remaining time threshold, wherein the remaining time threshold can be preset and is greater than 0. When the difference between the lag time and the data retention time of the message queue is less than or equal to the remaining time threshold, the data retention time of the message queue is adjusted.

[0069] The adjustment unit 130 adjusts the data retention time according to the maximum data retention time in the message queue, wherein the maximum data retention time is the longest time that the data in the message queue can be retained. In some scenarios, the sum of the lag time and the data retention time of the message queue is greater than or equal to the maximum data retention time of the message queue, and the adjustment unit 130 adjusts the data retention time to the maximum data retention time. In other scenarios, the sum of the lag time and the data retention time of the message queue is less than the maximum data retention time of the message queue, and the adjustment unit 130 adjusts the data retention time to the sum of the data retention time and the lag time. Thus, the adjustment unit 130 can extend the data retention time of the message queue, thereby preventing the clearing mechanism of Kafka from clearing data in the message queue that has not been consumed by the data consumer.

[0070] The monitoring system 100 is introduced below in conjunction with the alarm unit 140 and the condition unit 150 .

[0071] The alarm unit 140 can determine the relationship between the lag time and the alarm time threshold. The alarm time threshold can be pre-set. For example, the alarm time threshold can be less than or equal to the above-mentioned lag time threshold. Specifically, the alarm unit 140 feeds back an alarm message to the data consumer when the lag time is greater than or equal to the alarm time threshold. The alarm message can be used to prompt the data consumer that the lag time is too long. In order to enable the data consumer to obtain its own consumption progress, the alarm information also includes the data consumer's consumption progress, so that the data consumer can obtain its own consumption progress and then perform emergency processing, such as speeding up the consumption of data.

[0072] In some implementations, the hysteresis unit 120 is further configured to determine the amount of hysteresis data of the data consumer according to the offset information and the consumption progress. Figure 2 Take the offset information shown in as an example, if the consumption progress indicates that the data offset that the data consumer has consumed is 2000, if the current time is 4:00, then the offset of the data corresponding to the current time is 12600, and the lag unit 120 can determine the amount of lagged data by subtracting, that is, the amount of lagged data obtained by 12600-2000 in this embodiment is 12400. The alarm unit 140 can determine the relationship between the amount of lagged data and the alarm data amount threshold, and the alarm data amount threshold can be set in advance. When the amount of lagged data is greater than or equal to the alarm data amount threshold, the alarm unit 140 feeds back alarm information to the data consumer. In order to enable the data consumer to obtain its own consumption progress, the alarm information also includes the consumption progress of the data consumer, so that the data consumer can obtain its own consumption progress and then perform emergency processing, such as speeding up the consumption of data.

[0073] In some scenarios, different data consumers have different sensitivities to different types of alarms. For example, data consumer A is more sensitive to the lag time of data in the message queue, and data consumer B is more sensitive to the amount of lagged data in the message queue. Therefore, it is necessary to determine different alarm conditions based on the alarm configuration information of different data consumers. The alarm configuration information is used to describe the sensitivity of the data consumer to different types of alarms. Therefore, the communication unit 110 is also used to obtain the alarm configuration information of the data consumer. The condition unit 150 generates an alarm condition based on the alarm configuration information, and the alarm condition is used to trigger the above-mentioned alarm unit 140 to issue an alarm when it is met. For example, when the alarm condition is met, the alarm unit 140 feeds back the alarm information to the data consumer.

[0074] Next, in order to make the technical solution of the present application clearer and easier to understand, the method for data processing in the message queue provided in the embodiment of the present application is described in detail from the perspective of the monitoring system 100.

[0075] See also Figure 3 The flowchart of the method for processing data in the message queue shown in FIG. 1 comprises:

[0076] S302: The monitoring system 100 obtains the offset information of the data in the message queue of the target topic and the consumption progress of the data consumer for the data in the message queue of the target topic.

[0077] The offset information includes the offset of the data in the message queue within at least one statistical period. For details, see Figure 2 . The consumption progress includes the offset of the data that the data consumer has consumed in the message queue. For example, the consumption progress may be the offset of the data currently consumed by the data consumer. The present application does not limit the order in which the monitoring system 100 obtains the offset information and the consumption progress. In some implementations, the monitoring system 100 may periodically obtain the offset information and then obtain the consumption progress. For example: the monitoring system 100 obtains the offset of the data in the message queue at 1:30, the monitoring system 100 obtains the offset of the data in the message queue at 1:35... the monitoring system 100 obtains the offset of the data in the message queue at 4:00. That is, the monitoring system 100 counts the offset of the data in the message queue every 5 minutes. If the lag time needs to be determined at 4:03, the monitoring system 100 determines the lag time based on the consumption progress, the 4:00 time, and the offset of the data in the message queue counted before 4:00.

[0078] S304: The monitoring system 100 determines the lag time of the data consumer according to the offset information and the consumption progress.

[0079] As mentioned above, the monitoring system 100 counts the offset of the message queue data every 5 minutes, and then saves the offset of each statistical period to form offset information, which is convenient for subsequent determination of the lag time. Then, when the lag time needs to be determined, the offset information and consumption progress can be directly called to determine the lag time. Figure 2 Taking the offset information shown in as an example, if the consumption progress indicates that the offset of the data consumed by the data consumer is 2000, then it can be determined that the statistical time of the data is 1:30. If the current time is 4:00, it can be determined that the lag time is 150 minutes.

[0080] S306: The monitoring system 100 adjusts the data storage time of the message queue according to the lag time.

[0081] Due to the cleaning mechanism of Kafka itself, when the lag time of the data in the message queue exceeds the data retention time, the data in the message queue will be cleared. In order to avoid the data being cleared in this application, before the lag time exceeds the data retention time, the data retention time is adjusted, for example, the data retention time is delayed, thereby avoiding the situation where the data in the message queue is cleared. Thus, the monitoring system 100 can reduce the risk of data loss.

[0082] In some implementations, the monitoring system 100 can determine a relationship between a lag time and a lag time threshold, wherein the lag time threshold can be preset and is less than the data retention time. When the lag time is greater than or equal to the lag time threshold, the data retention time of the message queue is adjusted.

[0083] In other implementations, the monitoring system 100 can determine the relationship between the lag time and the remaining time threshold, wherein the remaining time threshold can be preset and is greater than 0. When the difference between the lag time and the data retention time of the message queue is less than or equal to the remaining time threshold, the data retention time of the message queue is adjusted.

[0084] The monitoring system 100 can determine different extension times according to different scenarios. The extension time refers to the time added on the basis of the data retention time of the message queue. The monitoring system 100 can also set a maximum data retention time, which indicates the longest time that the data in the message queue can be retained. After the lag time exceeds the maximum data retention time, the monitoring system 100 can no longer adjust the data retention time.

[0085] In some scenarios, the sum of the lag time and the data retention time of the message queue is greater than or equal to the maximum data retention time of the message queue, and the monitoring system 100 adjusts the data retention time to the maximum data retention time. In other scenarios, the sum of the lag time and the data retention time of the message queue is less than the maximum data retention time of the message queue, and the monitoring system 100 adjusts the data retention time to the sum of the data retention time and the lag time. As a result, the monitoring system 100 can extend the data retention time of the message queue, thereby preventing the Kafka clearing mechanism from clearing data in the message queue that has not been consumed by the data consumer.

[0086] The method obtains the offset information of the data in the message queue of the target topic and the consumption progress of the data consumer for the data in the message queue of the target topic. Since the offset information includes the offset of the data in the message queue within at least one statistical period, and the message progress includes the offset of the data already consumed by the data consumer in the message queue, the method can determine the lag time of the data consumer based on the offset information and the consumption progress, that is, the lag time is the lag time between the data in the message queue being consumed by the data consumer and the latest data in the message queue, and then adjust the data storage time of the message queue based on the lag time, thereby avoiding the situation where the data in the message queue is cleared due to exceeding the data storage time. Therefore, the method can reduce the risk of data loss.

[0087] The above describes how the monitoring system 100 adjusts the data retention time of the message queue. In some scenarios, the monitoring system 100 may also send warning information to the data consumer. Figure 4 The flowchart of the method for processing data in the message queue shown in FIG. 1 further includes, based on S302, S304 and S306:

[0088] S308: The monitoring system 100 obtains the alarm configuration information of the data consumer.

[0089] Different data consumers have different sensitivities to different alarm types. For example, data consumer A is more sensitive to the lag time of data in the message queue, while data consumer B is more sensitive to the amount of lag data in the message queue. Therefore, the monitoring system 100 needs to determine different alarm conditions based on the alarm configuration information of different data consumers.

[0090] S310: The monitoring system 100 generates an alarm condition according to the alarm configuration information.

[0091] After the monitoring system 100 obtains the alarm configuration information, it can generate an alarm condition according to the alarm configuration information. For example, if the alarm configuration information indicates that the data consumer is highly sensitive to the lag time, a condition related to the lag time is generated; if the alarm configuration information indicates that the data consumer is highly sensitive to the amount of lagged data, a condition related to the amount of lagged data is generated.

[0092] S312: The monitoring system 100 generates an alarm according to the alarm condition.

[0093] The alarm condition may be that the lag time is greater than or equal to the alarm time threshold, or that the amount of lag data is greater than or equal to the alarm data amount threshold. In some scenarios, if the alarm condition is that the lag time is greater than or equal to the alarm time threshold, then when the lag time is greater than or equal to the alarm time threshold, the monitoring system 100 feeds back alarm information to the data consumer. In other scenarios, if the alarm condition is that the amount of lag data is greater than or equal to the alarm data amount threshold, then the monitoring system needs to determine the amount of lag data of the data consumer based on the offset information and the consumption progress, and when the amount of lag data is greater than or equal to the alarm data amount threshold, the alarm information is fed back to the data consumer. The alarm information may include consumption progress, etc.

[0094] It should be noted that, in some implementations, the above S308, S310 and S312 may be executed before S304, or before S306, and the above is only an illustrative flowchart.

[0095] In this embodiment, the monitoring system 100 can alert the data consumer so that the data consumer can obtain its own consumption progress and speed up the consumption of data in the message queue under the prompt of the alarm information. As a result, there is no need for the data consumer to actively send a consumption progress acquisition request to the server corresponding to Kafka to obtain the consumption progress, thereby reducing the access pressure of the server. Furthermore, the monitoring system 100 also supports the generation of different alarm conditions for different data consumers, which improves the adaptability of the monitoring system 100.

[0096] Combination of the above Figures 1 to 4 A method for processing data in a message queue provided in an embodiment of the present application is introduced. Next, a data processing device in a message queue provided in an embodiment of the present application and a computing device for implementing the function of the data processing device in a message queue are introduced in conjunction with the accompanying drawings.

[0097] like Figure 5 As shown, the embodiment of the present application further provides a device 700 for processing data in a message queue, and the device 700 is used to execute the aforementioned method for processing data in a message queue. The embodiment of the present application does not limit the division of the functional modules in the device 700, and the following exemplary division of a functional module is provided:

[0098] The data processing device 700 in the message queue includes an acquisition module 702 , a determination module 704 and an adjustment module 706 .

[0099] The acquisition module 702 is used to obtain the offset information of the data in the message queue of the target topic and the consumption progress of the data consumer for the data in the message queue of the target topic, wherein the offset information of the data includes the offset of the data in the message queue within at least one statistical period, and the consumption progress includes the offset of the data consumed by the data consumer in the message queue. The determination module 704 is used to determine the lag time of the data consumer according to the offset information and the consumption progress. The adjustment module 706 is used to adjust the data storage time of the message queue according to the lag time.

[0100] The data processing device 700 in the above message queue can be implemented by a computing device. Figure 6 A computing device is provided, such as Figure 6 As shown, the computing device 800 can be specifically used to implement the above Figure 5 The functions of the data processing device 700 in the message queue in the illustrated embodiment.

[0101] The computing device 800 includes a bus 801, a processor 802, and a memory 803. The processor 802 and the memory 803 communicate with each other via the bus 801.

[0102] The processor 802 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0103] The memory 803 may include a volatile memory, such as a random access memory (RAM). The memory 803 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD) or a solid state drive (SSD).

[0104] The memory 803 stores executable program codes, and the processor 802 executes the executable program codes to execute the aforementioned data processing method in the message queue. Specifically, the processor 802 executes the above program codes to obtain the offset information of the data in the message queue of the target subject and the consumption progress of the data consumer for the data in the message queue of the target subject, the offset information of the data includes the offset of the data in the message queue within at least one statistical period, and the consumption progress includes the offset of the data consumed by the data consumer in the message queue. Then the processor 802 determines the lag time of the data consumer based on the offset information and the consumption progress, and adjusts the data storage time of the message queue based on the lag time.

[0105] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the data processing method in the above-mentioned message queue.

[0106] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system embodiment described above is merely schematic, and the units and modules described as separate components may or may not be physically separated. In addition, some or all of the units and modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0107] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0108] The above are only preferred embodiments of the present application, and are not intended to limit the present application in any form. Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technician familiar with the art can use the above disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present application without departing from the scope of the technical solution of the present application, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application still falls within the scope of protection of the technical solution of the present application.

Claims

1. A method for processing data in a message queue, characterized in that: include: Obtain the offset information of the data in the message queue of the target topic and the consumption progress of the data in the message queue of the target topic by the data consumer, wherein the offset information of the data includes the offset of the data in the message queue within at least one statistical period and the statistical time corresponding to the statistical period, and the consumption progress includes the offset of the data consumed by the data consumer in the message queue; Determine the lag time of the data consumer according to the offset information and the consumption progress; According to the lag time, the data storage time of the message queue is adjusted.

2. The method according to claim 1, characterized in that The method further includes: when the lag time is greater than or equal to an alarm time threshold, feeding back alarm information to the data consumer.

3. The method according to claim 1, characterized in that The method further comprises: Determine the amount of lagged data of the data consumer according to the offset information and the consumption progress; When the amount of delayed data is greater than or equal to the alarm data amount threshold, alarm information is fed back to the data consumer.

4. The method according to claim 2, characterized in that: The method further comprises: Obtain the alarm configuration information of the data consumer; An alarm condition is generated according to the alarm configuration information, and the alarm condition is used to trigger an alarm process when it is met.

5. The method according to claim 1, characterized in that The step of adjusting the data storage time of the message queue according to the lag time includes: When the lag time is greater than or equal to the lag time threshold, adjusting the data storage time of the message queue; or, When the difference between the lag time and the data retention time of the message queue is lower than or equal to the remaining time threshold, the data retention time of the message queue is adjusted.

6. The method according to any one of claims 1 to 5, characterized in that: The adjusting the data storage time of the message queue includes: If the sum of the lag time and the data retention time of the message queue is greater than or equal to the maximum data retention time of the message queue, adjust the data retention time to the maximum data retention time; If the sum of the lag time and the data retention time of the message queue is less than the maximum data retention time of the message queue, adjust the data retention time to the sum of the data retention time and the lag time.

7. A data processing device in a message queue, characterized in that: include: Acquire modules, determine modules, and adjust modules; The acquisition module is used to acquire the offset information of the data in the message queue of the target topic and the consumption progress of the data consumer for the data in the message queue of the target topic, wherein the offset information of the data includes the offset of the data in the message queue within at least one statistical period and the statistical time corresponding to the statistical period, and the consumption progress includes the offset of the data consumed by the data consumer in the message queue; The determination module is used to determine the lag time of the data consumer according to the offset information and the consumption progress; The adjustment module is used to adjust the data storage time of the message queue according to the lag time.

8. The device according to claim 7, characterized in that The device also includes: an alarm module; The alarm module is used to feed back alarm information to the data consumer when the lag time is greater than or equal to the alarm time threshold.

9. A data processing device in a message queue, characterized in that: The device comprises: a memory and a processor; The memory is used to store a computer program and transmit the computer program to the processor; The processor executes the method for processing data in a message queue according to any one of claims 1 to 6 according to the instructions in the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer software instructions used by the data processing system in the message queue. When the computer-readable storage medium is run on a computer, the computer can execute the data processing method in the message queue described in any one of claims 1 to 6.

Images