Message accumulation amount statistical method and device, equipment and storage medium

Abstract

The invention provides a message accumulation amount statistical method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining a current consumption site of a message queue, and obtaining the expiration time corresponding to each time window; based on the current statistical moment, the current consumption site and the expiration time of each time window, determining a first time window in which the message accumulation amount needs to be counted again and a second time window in which the message accumulation amount does not need to be counted again from each time window; and calculating the message accumulation amount in the message queue at the current statistical moment based on the historical message accumulation amount of each second time window counted at the previous statistical moment and the newly counted message accumulation amount corresponding to each first time window. According to the invention, under the condition that expired messages exist, the statistical difficulty of the message accumulation number can be effectively reduced, and the statistical speed and accuracy of the message accumulation number can be improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and specifically relates to a message accumulation statistics method, device, equipment and storage medium. Background Art

[0002] A message is a unit of data transmitted between two computers. A message queue (MQ) is a container that stores messages during the transmission process between two computers. It is widely used in Internet technology storage.

[0003] For message queues, message accumulation (backlog) is a common problem. Message accumulation means that messages are not consumed in time. It is caused by the producer producing messages faster than the consumer consuming them. In actual application, the number of accumulated messages can be counted to monitor the status of the message queue. For example, an early warning can be issued when the number of accumulated messages is too large.

[0004] At present, in order to expand more usage scenarios, some message protocols can support message expiration features (messages will be discarded after a certain period of time). However, this greatly increases the difficulty of counting the number of accumulated messages. Summary of the invention

[0005] The present application proposes a message accumulation amount statistics method, device, equipment and storage medium, which can effectively reduce the difficulty of counting the number of message accumulations when there are expired messages, and improve the counting speed and accuracy of the number of message accumulations.

[0006] The first embodiment of the present application proposes a message accumulation amount statistics method, including:

[0007] Obtain the current consumption position of the message queue, and respectively obtain the expiration time corresponding to each time window, where the expiration time corresponding to the time window is the expiration time of all messages in the message queue whose timestamps are within the time window;

[0008] Based on the current statistical moment, the current consumption location and the expiration time of each time window, determine from the time windows a first time window for which message accumulation needs to be re-stated and a second time window for which re-statistics is not required;

[0009] The message accumulation amount in the message queue at the current statistical moment is calculated based on the historical message accumulation amount of each second time window counted at the previous statistical moment and the message accumulation amount corresponding to each first time window counted newly.

[0010] In some embodiments of the present application, based on the current statistical moment, the current consumption location, and the expiration time of each time window, determining from each time window a first time window for which message accumulation amount needs to be re-counted and a second time window for which re-counting is not required, includes:

[0011] Determine the time interval between the current statistical moment and the previous statistical moment, and the target time window where the current consumption location is located;

[0012] The time window in which the minimum value of the expiration time is less than or equal to the time interval, the time period between the previous statistical moment and the current statistical moment, and the time period between the current consumption location and the end time of the target time window are all determined as the first time window;

[0013] The time windows other than the first time window in each time window are determined as the second time window.

[0014] In some embodiments of the present application, the amount of accumulated messages in the message queue at the current statistical moment is calculated based on the historical amount of accumulated messages in each second time window counted at the previous statistical moment and the amount of accumulated messages corresponding to each first time window counted newly, including:

[0015] Determine, according to the start time and the end time of each of the first time windows, the messages whose timestamps are within each of the first time windows;

[0016] Based on the message identifiers of the messages in each of the first time windows, recounting the message accumulation amount in each of the first time windows; the message identifiers at least include a timestamp, a sequence number, and a generating device identifier;

[0017] The historical message accumulation amount of each second time window and the re-counted message accumulation amount of each first time window are summed to obtain the message accumulation amount in the message queue at the current statistical moment.

[0018] In some embodiments of the present application, the re-counting of message accumulation amounts in each of the first time windows based on message identifiers of messages in each of the first time windows includes:

[0019] Determine a message identification range of the messages in each of the first time windows based on the message identifications of the messages in each of the first time windows; the message identification range includes a timestamp range, a sequence number range, and a generating device identification range of the messages;

[0020] According to the generation rule of the message identifiers and the message identifier ranges of the messages in each of the first time windows, the message accumulation amount in each of the first time windows is re-counted.

[0021] In some embodiments of the present application, the method further includes:

[0022] Upon receiving a new message, insert the new message into the end of the message queue;

[0023] Determine whether the timestamp of the new message reaches the end time of the current time window;

[0024] If so, the message accumulation amount in the current time window is counted, and the counted message accumulation amount in the current time window is stored.

[0025] In some embodiments of the present application, the method further includes:

[0026] When the first time window for re-counting the message accumulation amount appears, the currently stored historical message accumulation amount of the first time window is updated to the re-counted message accumulation amount.

[0027] In some embodiments of the present application, obtaining the current consumption position of the message queue includes:

[0028] When the user resets the consumption location at the current statistical moment, the reset consumption location is determined as the current consumption location;

[0029] If the user has not reset the consumption location at the current statistical moment, the time corresponding to the message with the earliest timestamp among the unconsumed messages is determined as the current consumption location.

[0030] The second embodiment of the present application provides a message accumulation amount statistics device, including:

[0031] A message information acquisition module is used to obtain the current consumption position of the message queue and respectively obtain the expiration time corresponding to each time window. The expiration time corresponding to the time window is the expiration time of all messages in the message queue whose timestamps are within the time window.

[0032] A time window determination module, configured to determine, from each time window, a first time window for which message accumulation needs to be re-counted and a second time window for which re-counting is not required, based on the current statistical moment, the current consumption location and the minimum expiration time of each time window;

[0033] The accumulation amount calculation module is used to calculate the message accumulation amount in the message queue at the current statistical moment based on the historical message accumulation amount of each second time window counted at the previous statistical moment and the message accumulation amount corresponding to each first time window counted newly.

[0034] An embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0035] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect above.

[0036] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0037] In an embodiment of the present application, based on the idea of ​​segmented statistics, the historical message accumulation amount corresponding to each time window, the current consumption position of the message queue, and the expiration time corresponding to each time window are first obtained, and based on the expiration time of each window, the current statistical time and the current consumption position, the first time window that needs to re-count the message accumulation amount and the second time window that does not need to be re-counted are determined, and then the current message accumulation amount corresponding to the second time window can be directly adopted The previous statistical result; the current message accumulation amount corresponding to the first time window can be re-counted, and based on the historical message accumulation amount corresponding to each second window and the re-counted message accumulation amount corresponding to each first time window, the message accumulation amount in the message queue at the current statistical moment is determined, thereby realizing a low-cost message accumulation amount statistical method, which can well support the message expiration characteristics and improve the statistical accuracy and speed of the message accumulation amount.

[0038] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0040] In the attached picture:

[0041] Figure 1 A schematic diagram of primary keys stored in a message queue in some embodiments of the present application is shown;

[0042] Figure 2 A schematic diagram showing a message queue in which a consumption site changes in some embodiments of the present application;

[0043] Figure 3A schematic diagram showing an application scenario of a message accumulation amount statistics method provided by some embodiments of the present application is shown;

[0044] Figure 4 A schematic diagram of a flow chart of a message accumulation amount counting method provided in some embodiments of the present application is shown;

[0045] Figure 5 A schematic diagram of a message queue including multiple time windows at time T in an embodiment of the present application is shown;

[0046] Figure 6 A schematic diagram of a message queue including multiple time windows at time T+1 in an embodiment of the present application is shown;

[0047] Figure 7 A schematic diagram of the specific process in step S2 in an embodiment of the present application is shown;

[0048] Figure 8 A schematic diagram of the specific process in step S3 in an embodiment of the present application is shown;

[0049] Fig. 9 A schematic diagram showing the composition of a message ID in an embodiment of the present application is shown;

[0050] Fig.10 A schematic diagram of a message queue with expired messages in one embodiment of the present application is shown;

[0051] Fig.11 A schematic diagram showing the structure of a message accumulation amount counting device provided in some embodiments of the present application is shown;

[0052] Fig.12 A schematic diagram of an electronic device provided by an embodiment of the present application is shown.

[0053] Fig.13 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0054] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0055] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0056] A message is a unit of data transmitted between two computers. The data structure of a message can be very simple, such as containing only text strings, or more complex, such as possibly containing embedded objects. Service data can also be encapsulated in the form of data packets to achieve message transmission. Among them, data packets can be encoded in a fixed format defined by different communication protocols. A queue is a first-in-first-out (FIFO) data structure. A message queue (MQ) can also be called a message middleware. It can be seen as a container for storing messages during message transmission. It is an important component in a distributed system. The main purpose of using a message queue is to improve system performance, reduce peaks, and reduce system coupling through asynchronous processing.

[0057] The number of messages stored in a message queue is related to the speed at which producers send or publish messages to the message queue and the speed at which consumers consume messages from the message queue. When the producer sends messages faster than the consumer consumes messages, such as when there are too many concurrent accesses, messages will accumulate in the message queue. However, the capacity and accumulation capacity of a message queue are limited. If the amount of accumulated messages is too large, it will have an adverse effect on the performance of the server storing the messages, such as large data processing delays, reduced service quality, and even downtime. Therefore, in actual applications, the number of accumulated messages, that is, the number of messages in the current message queue, can be counted to monitor the status of the message queue, and an early warning can be issued when the number of accumulated messages is too large.

[0058] When counting the amount of message accumulation, how to quickly and accurately count the amount of message accumulation in the message queue is a problem that has long been to be solved by industry insiders. Moreover, some current message transmission protocols, such as but not limited to MQTT (Message Queuing Telemetry Transport), can support message expiration features, that is, a lifetime can be set for messages in the message queue. If the storage time of the message in the message queue exceeds the lifetime, it is called an expired message. The system can directly discard the expired message to reduce the amount of message accumulation. In the case where the message queue supports the message expiration feature, the amount of message accumulation is not only related to the number of messages generated and the number of messages consumed, but also to the amount of expired messages, which makes the statistics of the message accumulation more difficult.

[0059] In the related art, for application scenarios with large amounts of data, such as but not limited to machine-to-machine (M2M) communication and Internet of Things (IoT) platforms, the message accumulation statistics can be implemented based on SLS (Simple Log Service, log management service), which is called the counting method. Specifically, when the producer sends a message, a specific log is printed, and when the consumer consumes a message, another log is printed. When the final statistics are made, the total number of sent logs and the total number of consumed logs are counted, and the difference between the two is the message accumulation. However, there is a delay in the SLS log itself, and the accuracy is poor when counting a large number of logs. There is often a large error between the accumulated amount and the actual message accumulation. As time goes by, the accuracy of the count will become lower and lower. For a message queue that supports message expiration, if the message is valid for only n minutes, then when counting the message accumulation within n minutes, the currently expired message needs to be subtracted, but the method using a counter cannot identify and exclude expired messages. In addition, when the amount of message sending / consumption is particularly large, the counting method will have a hot spot problem (during the counting process, some key values ​​are counted more frequently than other key values. These key values ​​are the so-called hot data. When processing requests in a single thread, the efficiency of processing hot data is often low and can easily become a bottleneck of the system).

[0060] In order to solve the problems encountered by the above counting method in the process of counting the message accumulation, the relevant technology also adopts the big data statistics method, importing the data into the data warehouse, such as but not limited to MaxCompute (big data computing service), and analyzing the massive data through the data warehouse's capabilities to count the message accumulation. However, if the message accumulation statistics are not in the runtime link, the frequency of use of this function is not high. Importing all messages into MaxCompute just to count the accumulation is too costly. In addition, the Internet of Things platform supports the accumulation of billions of messages. If the data warehouse counts the accumulation based on billions of data, it will not only consume a lot of computing resources, but also affect the overall performance and reduce service stability.

[0061] The present application embodiment studies the current message accumulation statistics method and finds that the information about the message can be stored in a table, but is not limited to it. For the consumer queue, the primary key of the stored table can be the consumer ID and the message ID, such as Figure 1As shown. For the producer queue, the primary key of the stored table can be the producer ID and the message ID. And as the number of messages increases, the timestamp of the message also increases. When counting the message accumulation in the message queue, you can query and store the message table according to the consumer ID range to obtain a more accurate message accumulation. However, if the accumulation is too large, such as millions or even tens of millions of messages, directly counting the number of records in the table is very time-consuming, extremely consuming, and has extremely poor performance. In response to this problem, the embodiment of the present application first proposes a segmentation idea, that is, caching the message accumulation at the previous statistical moment (T moment), and at the current statistical moment, it is recorded as T+1 moment. Without considering the change and expiration of the consumption site, it is only necessary to calculate the message accumulation from T moment to T+1 moment, and the message accumulation before T moment can directly use the cached data. However, the consumption site often changes between two statistical moments, such as Figure 2 As shown, at time T, the consumption point is at A. At time T+1, the consumption point may not be at A but at B. Moreover, expired messages may be generated from time T to time T+1. Therefore, directly using the previously cached message accumulation before time T is no longer accurate.

[0062] Based on the above reasons, an embodiment of the present application provides a method for counting the amount of message accumulation. Based on the timing of the message queue, the message queue is regarded as a window queue composed of multiple time windows. Messages within the corresponding time period are stored in each time window. When the message accumulation needs to be counted, the current consumption point of the message queue can be obtained first, and the expiration time corresponding to each time window can be obtained respectively; then, based on the current statistical moment, the current consumption point and the expiration time of each time window, the first time window for which the message accumulation needs to be re-counted and the second time window that does not need to be re-counted are determined from each time window; and based on the historical message accumulation of each second time window counted at the previous statistical moment, and the message accumulation corresponding to each newly counted first time window, the message accumulation in the message queue at the current statistical moment is calculated. In this way, the idea of ​​segmented statistics is adopted to obtain the historical message accumulation amount corresponding to each time window, and based on the expiration time of each window, the current statistical time and the current consumption location, the first time window that needs to re-count the message accumulation amount and the second time window that does not need to be re-counted are determined, and then the current message accumulation amount corresponding to the second time window can be directly adopted The previous statistical result; the current message accumulation amount corresponding to the first time window can be re-counted, and based on the historical message accumulation amount corresponding to each second window and the re-counted message accumulation amount corresponding to each first time window, the message accumulation amount in the message queue at the current statistical moment is determined, thereby realizing a low-cost message accumulation amount statistics method, which can well support the message expiration characteristics and improve the statistical accuracy and speed of the message accumulation amount.

[0063] Among them, the expiration time corresponding to the time window is the expiration time of all messages in the message queue whose timestamps are within the time window. The timestamp in the message queue can represent the generation time of each message. The consumption site can be understood as the location of the message to be consumed in the message queue, which can correspond to a timestamp. The previous statistical moment can be understood as the time when the overall accumulation of the message queue was last counted. The time difference between each current statistical moment and its previous statistical moment can be the same or different. The important thing is to record each statistical moment. The first time window in which the message accumulation needs to be re-counted may include a time window in which the message accumulation changes between the current statistical moment and the previous statistical moment, such as a time window with expired messages, a time window where the consumption site is located, and a time window where the current statistical moment is located.

[0064] The execution subject of the message backlog statistics method may be a third party such as a computer, server, server cluster, cloud platform, etc. that provides message queues and performs message management, or it may be a message backlog statistics device that specifically executes the message backlog statistics method on these devices. This embodiment does not specifically limit this. It can be understood that the execution subject may be neither a producer nor a consumer of messages, and it may act as a third party to manage message queues, i.e., messages in queues. Taking the execution subject as a cloud platform as an example, the message backlog statistics method can be applied to, for example, Figure 3 In the scenario shown, the cloud platform provides a message queue, the producer client produces / publishes messages and sends the messages to the message queue, the consumer client can obtain and consume messages from the message queue, the cloud platform can monitor the status of the message queue in real time, regularly or irregularly count the amount of message accumulation in the message queue, and issue an early warning when the amount of message accumulation is too large.

[0065] The message accumulation statistics method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0066] See also Figure 4 , is a flow chart of a message accumulation amount statistics method provided in an embodiment of the present application, such as Figure 4 As shown, the message accumulation amount statistics method may include the following steps:

[0067] Step S1, obtaining the current consumption position of the message queue, and respectively obtaining the expiration time corresponding to each time window.

[0068] Among them, a message queue can be used to store messages sent / published by the same producer or by multiple different producers, as long as the stored messages are sorted by time based on timestamps. The messages stored in a message queue can correspond to one consumer or multiple different consumers, as long as the messages are consumed in order of the timestamps. For each message queue, Figure 5 As shown, the message queue can be divided into multiple time windows, and the duration corresponding to each time window can be the same or different, as long as the start time and end time of each time window are recorded. The current consumption position of the message queue can be the same as the consumption position at the previous statistics, or it can be different. If the current consumption position is the same as the consumption position at the previous statistics, when the message accumulation amount of each time window is subsequently counted, only the current message accumulation amount of the time window at the current moment needs to be counted, and the message accumulation amount corresponding to other time windows can directly use the previous statistical results. If the consumption position changes, when the message accumulation amount of each time window is subsequently counted, not only the current message accumulation amount of the time window at the current moment needs to be counted, but also the message accumulation amount corresponding to the time window at the current consumption position needs to be counted. In addition, since the message queue supports the message expiration feature, between the previous statistical moment and the current statistical moment, there may be expired messages in the messages corresponding to some time windows, and the expired messages will be directly discarded, so the message accumulation amount corresponding to the time window will be changed, and the message accumulation amount corresponding to the time window with expired messages needs to be re-counted. The expiration time corresponding to the time window is the expiration time of all messages in the message queue whose timestamps are within the time window.

[0069] In some embodiments, the user can reset the consumption location by himself. When obtaining the current consumption location of the message queue, it can be determined first whether the user has reset the consumption location, and if the user resets the consumption location at the current statistical moment, the reset consumption location is determined as the current consumption location; if the user has not reset the consumption location at the current statistical moment, the time corresponding to the message with the earliest timestamp among the unconsumed messages is determined as the current consumption location.

[0070] The user can be any party that can use and manage the message queue, and can be a message producer or a message consumer, as long as they have the authority to use and manage the message queue. The user resetting the consumption position at the current statistical moment can be understood as the user resetting the consumption position before the current statistical moment, and the time corresponding to the reset consumption position is later than the earliest time of the message stored in the current message queue. Figure 5 and Figure 6As shown, assuming that the consumption point at the previous statistical moment (T moment) is A, and the consumption point at the current statistical moment (T+1 moment) is D, when counting the total message accumulation at the current statistical moment, the messages before the consumption point D are counted regardless of whether they are consumed.

[0071] In practical applications, accurately and quickly determining the current consumption location is a prerequisite for accurately and quickly counting the current message accumulation amount. In this embodiment, when the user resets the consumption location at the current statistical moment, the reset consumption location is directly used without calculation, which can further improve the statistical time of the message accumulation amount. And when the user does not reset the consumption location at the current statistical moment, the time corresponding to the message with the earliest timestamp among the unconsumed messages is determined as the current consumption location, so that the current consumption location can be accurately determined, thereby improving the statistical accuracy of the message accumulation amount.

[0072] Step S2, based on the current statistical time, the current consumption location and the expiration time of each time window, determine from each time window a first time window in which message accumulation amount needs to be re-stated and a second time window in which re-statistics is not required.

[0073] Among them, the current statistical moment can be any current moment, which can be located in any time window. The message accumulation counted at the current statistical moment can be understood as the message accumulation amount of each time window at the current moment of the message queue whose tail corresponds to the current statistical moment. The above-mentioned "re-statistics" can include updates to the original data, and can also include statistics on new data, that is, the first time window for re-stating the message accumulation amount, that is, it can include the time window in which the message accumulation amount has been counted before, and the previous statistical results are inaccurate due to the consumption site and expiration characteristics, and need to be re-counted; it can also include the time window in which the current statistical moment is located. The time interval of this time window may be after the previous statistical moment, so the message accumulation amount has not been counted before, and the current re-statistics can also be understood as the previous statistical result being 0.

[0074] Based on the above principle of determining the consumption site, it can be known that the consumption site is not necessarily located at the start or end time of a time window, so the time window where the consumption site is located often needs to re-count the message accumulation, so the time window where the consumption site is located is the first time window. Similarly, the current statistical moment is not necessarily located at the start or end time of a time window, that is, the time window where the current statistical moment is located also needs to re-count the message accumulation, and the time window where the current statistical moment is located is also the first time window. This embodiment also takes into account that the message queue may support the message expiration feature, and some time windows may generate expired messages between the previous statistics and the current statistical moment, so even if there are no new messages or new consumption messages in these time windows, the message accumulation will still change, and the message accumulation needs to be re-counted, and these windows also belong to the first time window. In this way, the time window related to the message queue and the current message accumulation is divided into the first time window where the message accumulation needs to be re-counted, and the second time window where the message accumulation does not need to be newly counted, which takes into account the generation of new messages and the consumption of original messages, as well as the expiration characteristics of messages, so that the final statistical results can be more accurate.

[0075] In some embodiments, Figure 7 As shown, the above step S2 may further include the following steps: step S21, determining the time interval between the current statistical moment and the previous statistical moment, and the target time window where the current consumption location is located; step S22, determining the time window whose minimum expiration time is less than or equal to the time interval, the time period between the previous statistical moment and the current statistical moment, and the time period between the current consumption location and the end time of the target time window as the first time window; step S23, determining the time window except the first time window in each time window as the second time window.

[0076] In this embodiment, when determining the first time window and the second time window, first determine whether there are expired messages in the messages in each time window within the time interval based on the time interval between the current statistical moment and the previous statistical moment, and the expiration time of each time window, and determine the time window with expired messages within the time interval as the first time window. And the time period between the end time of the target time window where the current consumption site is located and the current consumption location, and the time period between the previous statistical moment and the current statistical moment are all determined as the first time window, so that the factors affecting the message accumulation amount, such as the consumption site, the message expiration characteristics, the current statistical moment, etc., are fully considered, so that the determination of the first time window that needs to re-count the message accumulation amount is more accurate.

[0077] Specifically, when determining the target time window where the current consumption location is located, you can first determine the timestamp of the current consumption location, as well as the start time and end time of each time window; then based on the start time and end time of each time window, you can quickly determine the time window to which the timestamp of the current consumption location belongs, and determine the time window as the time window where the current consumption location is located, so that you can quickly and accurately determine the target time window where the current consumption location is located.

[0078] Step S3, based on the historical message accumulation amount of each second time window counted at the previous statistical moment and the message accumulation amount corresponding to each first time window counted newly, the message accumulation amount in the message queue at the current statistical moment is calculated.

[0079] The historical message accumulation amount can be understood as the message accumulation amount counted at the last statistical moment.

[0080] After determining the first time window and the second time window, this embodiment can respectively determine the message accumulation amounts corresponding to the first time window and the second time window, and the message accumulation amount corresponding to the first time window needs to be recalculated, while the message accumulation amount corresponding to the second time window does not need to be recalculated, and only the previous statistical results can be used, thereby improving the calculation speed of the statistical message accumulation amount.

[0081] In some embodiments, Figure 8 As shown, the above step S3 may further include the following steps: step S31, determining the messages whose timestamps are within each first time window based on the start time and end time of each first time window; step S32, re-counting the message accumulation amount of each first time window based on the message identifier of the message within each first time window; step S33, summing the historical message accumulation amount of each second time window and the re-counted message accumulation amount of each first time window to obtain the message accumulation amount in the message queue at the current statistical moment.

[0082] The above-mentioned message identifier at least includes a timestamp, a serial number and a generating device identifier. Among them, the timestamp can be counted in milliseconds, and can be, but not limited to, a 41-bit string. This embodiment can be but not limited to using a Java timestamp, that is, recording the total number of milliseconds from Greenwich Mean Time 00:00:00 on January 1, 1970 (Beijing Time 08:00:00 on January 1, 1970) to the present (excluding leap seconds). 41 bits can be reserved to save the millisecond timestamp, because the length of the millisecond time is 13 bits, and the maximum value of the 41-bit binary maximum value is (2 41-1=2199023255551, exactly 13 bits. The sequence number can be a serial number within a millisecond, which can increase as the number of messages generated by the device increases, and can be, but not limited to, a 9-bit or 12-bit string. The generating device identifier can be a string that can identify the message producer, and can be, but not limited to, a 10-bit or 13-bit string. The generating device can be, but not limited to, an IoT platform, an IoT platform, etc. In addition, the message identifier can also include a sign bit, which can usually be set to 0. Based on the first-in-first-out principle of the message queue, the timestamp of the message queue has a certain interval, and the two extreme values ​​of the timestamp correspond to the two ends of the message queue, that is, the smallest timestamp corresponds to the consumption position of the current message queue, and the largest timestamp corresponds to the end of the current message queue. When a consumer consumes a message, it consumes from the message with the earliest timestamp to the message with the later timestamp.

[0083] In actual applications, each message generated by a producer can have a unique identifier, namely, a message ID (Identity document), which is used to identify the message. The generation rule of the message ID is similar to the snowflake algorithm, using Fig. 9 The format of "timestamp + generating device identification + serial number" is shown. Based on the generation rule of message ID, it can be known that each message ID of the message queue can fall within a large time period, and the two end values ​​of the time period are the minimum and maximum values ​​of the timestamp in the message queue.

[0084] Assuming that the time period is 2022-10-01 00:00:00 to 2022-10-01 00:01:00, and the corresponding timestamp is 1664553600000 to 1664553660000, according to the above message ID generation rule, the generated message IDs all fall within the following range ((1664553600000<<22)|(0<<9)|0)≤MessageID<((1664553660000<<22)|(0<<9)|0). Among them, << means that the data on the left is obtained by shifting the data on the right to the left. For example, if a message is generated at 2022-10-01 00:00:30, the timestamp is 1664553630000. Assuming the machine ID is 5 and the sequence number is 10, the message ID = ((1664553630000<<22)|(5<<9)|10). It can be seen that (1664553630000<<22)|(5<<9)|10) is indeed in the interval ((1664553600000<<22)|(0<<9)|0)≤Message ID<((1664553660000<<22)|(0<<9)|0).

[0085] Based on the above message ID generation rules, it is easy to obtain the message ID of each message in each time window, and then based on the message ID in each time window, the message accumulation amount corresponding to each first time window can be quickly and accurately re-counted, and then the message accumulation amount of each first time window re-counted can be summed up to obtain the message accumulation amount in the message queue at the current statistical moment.

[0086] It is understandable that this embodiment only limits the components of the message ID, and does not limit the order of composition of the components, that is, the message ID is not limited to the order of "timestamp+generating device identifier+serial number", as long as it includes these three parts.

[0087] Specifically, the above step S32 may include the following processing: based on the message identifiers of the messages in each first time window, determine the message identifier range of the messages in each first time window; according to the message identifier generation rules and the message identifier range of the messages in each first time window, re-count the message accumulation amount in each first time window.

[0088] The message representation range is the range of the message ID, including the timestamp range, sequence number range and generating device identification range of the message, which can also be understood as the interval of (minimum message ID) ≤ message ID < (maximum message ID).

[0089] In this embodiment, when calculating the message accumulation amount of each first time window, for the first time window where the current statistical moment is located and other first time windows without expired messages, the messages within the message ID interval of each first time window can be directly counted.

[0090] For the target time window where the consumption site is located, the timestamp corresponding to the consumption site and the timestamp corresponding to the end time of the target time window can be determined first according to the message ID corresponding to the consumption site, and then the message accumulation amount between the timestamp corresponding to the consumption site and the timestamp corresponding to the end time of the target time window can be counted. Then determine the timestamp corresponding to the end time of the target time window, that is, determine the latest end time after the timestamp corresponding to the consumption site. Assuming that the time interval between the previous statistical moment and the current statistical moment is one minute, the message ID corresponding to the consumption site is recorded as messageId, and the millisecond timestamp is recorded as timestamp, then timestamp = messageId>>22 (>> means removing some bits of data on the right), the number of minutes is minute = timestamp / 1000 / 60, then the timestamp corresponding to the latest end time after the timestamp corresponding to the consumption site is latestTimestamp = (minute+1)*1000*60.

[0091] As mentioned above, after determining the timestamp corresponding to the consumption site and the timestamp corresponding to the end of the target time window, the number of messages within the time interval formed by the two timestamps can be counted, that is, the amount of message accumulation that needs to be re-counted in the target time window where the consumption site is located.

[0092] In addition, when caching the message accumulation corresponding to each time window, the expiration time min (expiry interval) of all messages in this window can be recorded at the same time. In this way, the timeliness of the message accumulation in each time window can be determined based on the minimum value of the expiration time of all messages in the time window, and then the first time window with expired messages in the message queue can be determined, such as Fig.10 The darker time window. If there are expired messages in a time window, the message accumulation corresponding to the time window is re-counted; for the time window without expired messages, the cached message accumulation at time T can be reused at time T+1. In this way, even if the message has an expiration time, the message accumulation can still be counted quickly and accurately.

[0093] In other embodiments, based on the message queue divided into time windows, after receiving a new message, the message accumulation amount statistics device can first insert the new message into the end of the corresponding message queue, and then determine whether the timestamp of the new message reaches the end time of the current time window. If it reaches the end time of the current time window, the message accumulation amount of the current time window is counted and the counted message accumulation amount of the current time window is stored.

[0094] The current time window can be understood as the time window whose end time is closest to the time when the new message is sent to the message queue. Specifically, if the time when the new message is sent to the message queue is very close to or just reaches the end time of a certain time window, then the time window is the current time window. Very close here can be slightly less than the end time of a certain time window, or slightly greater than the end time of a certain time window.

[0095] In this embodiment, the start time and end time of each time window can be preset, and after reaching the end time of each time window, the message accumulation amount of the time window that currently reaches the end time can be counted. Therefore, when a new message is sent to the message queue, the new message can be first inserted into the end of the message queue according to the first-in-first-out principle, and then it is determined whether the current time reaches the end time of the current time window. If the end time of the current time window is reached, the message accumulation amount of the current time window is re-counted, and the message accumulation amount of the current time window is stored so as to calculate the message accumulation amount of the message queue at the current statistical moment. If the end time of the current time window is not reached, the message accumulation amount of the current time window can be temporarily not counted. After the message queue receives a new message again, it is determined again whether the timestamp of the new message reaches the end time of the current time window.

[0096] Furthermore, the message accumulation amount statistics method may also include the following processing: when the first time window for re-counting the message accumulation amount appears, the historical message accumulation amount of the first time window currently stored is updated to the re-counted message accumulation amount. In this way, the historical message accumulation amount of each time window is updated in real time, so that when the message accumulation amount is counted again at the subsequent statistical moment, a more accurate historical message accumulation amount can be obtained in time, further improving the accuracy of the message accumulation amount statistics method.

[0097] In summary, the message accumulation amount statistics method provided in this embodiment is based on the idea of ​​segmented statistics. The historical message accumulation amount corresponding to each time window, the current consumption position of the message queue, and the expiration time corresponding to each time window are first obtained. Based on the expiration time of each window, the current statistical time and the current consumption position, the first time window that needs to re-count the message accumulation amount and the second time window that does not need to be re-counted are determined. Then, for the current message accumulation amount corresponding to the second time window, the previous statistical result can be directly adopted; the current message accumulation amount corresponding to the first time window can be re-counted, and based on the historical message accumulation amount corresponding to each second window and the message accumulation amount corresponding to each re-counted first time window, the message accumulation amount in the message queue at the current statistical moment is determined, thereby realizing a low-cost message accumulation amount statistics method, which can well support the message expiration characteristics and improve the statistical accuracy and speed of the message accumulation amount.

[0098] Some embodiments of the present application further provide a message accumulation amount counting device, which is used to execute the message accumulation amount counting method provided in any of the above embodiments, Fig.11 A schematic diagram of the message accumulation amount counting device is shown, as shown in Fig.11 As shown, the message accumulation amount counting device includes:

[0099] The message information acquisition module is used to obtain the current consumption position of the message queue and the expiration time corresponding to each time window. The expiration time corresponding to the time window is the expiration time of all messages in the message queue whose timestamps are within the time window.

[0100] A time window determination module is used to determine, from each time window, a first time window for which message accumulation needs to be re-counted and a second time window for which re-counting is not required, based on the current statistical moment, the current consumption location and the expiration time of each time window;

[0101] The accumulation amount calculation module is used to calculate the message accumulation amount in the message queue at the current statistical moment based on the historical message accumulation amount of each second time window counted at the previous statistical moment and the message accumulation amount corresponding to each first time window counted newly.

[0102] It can be understood that the message accumulation amount statistics device provided in this embodiment and the message accumulation amount statistics method provided in the embodiment of the present application are based on the same inventive concept, and can at least achieve the same beneficial effects of the message accumulation amount statistics method, and the various implementation methods of the message accumulation amount statistics method embodiment are also applicable to the embodiment of the message accumulation amount statistics device, which will not be repeated here.

[0103] It should be noted that the data involved in this application (including but not limited to data used for model training, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0104] The present application also provides an electronic device to execute the above message accumulation amount statistics method. Fig.12 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Fig.12 As shown, the electronic device 4 includes: a processor 400, a memory 401, a bus 402 and a communication interface 403, and the processor 400, the communication interface 403 and the memory 401 are connected via the bus 402; the memory 401 stores a computer program that can be run on the processor 400, and when the processor 400 runs the computer program, it executes the message accumulation quantity statistics method provided in any of the aforementioned embodiments of the present application.

[0105] The memory 401 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 403 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0106] The bus 402 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 401 is used to store programs, and the processor 400 executes the programs after receiving the execution instruction. The message accumulation amount statistics method disclosed in any implementation of the embodiment of the present application may be applied to the processor 400, or implemented by the processor 400.

[0107] The processor 400 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 400. The above processor 400 can be a general-purpose processor, including a processor (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 401, and the processor 400 reads the information in the memory 401 and completes the steps of the above method in combination with its hardware.

[0108] The electronic device provided in the embodiment of the present application and the message accumulation amount statistics method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0109] The present application also provides a computer-readable storage medium corresponding to the message accumulation amount statistics method provided in the above embodiment. Fig.13 The computer-readable storage medium shown is a CD 50 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the message accumulation amount statistics method provided by any of the aforementioned embodiments will be executed.

[0110] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0111] An embodiment of the present application also provides a computer program product, including a computer program, which is executed by a processor to implement the message accumulation amount statistics method described in any of the above embodiments.

[0112] The computer-readable storage medium and computer program product provided in the above-mentioned embodiments of the present application are based on the same inventive concept as the message accumulation quantity statistics method provided in the embodiments of the present application, and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0113] It should be noted that:

[0114] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0115] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features explicitly recited in the individual claims. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0116] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0117] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A message accumulation statistics method, characterized in that: include: Obtain the current consumption position of the message queue, and respectively obtain the expiration time corresponding to each time window, where the expiration time corresponding to the time window is the expiration time of all messages in the message queue whose timestamps are within the time window; Based on the current statistical moment, the current consumption location and the expiration time of each time window, determine from the time windows a first time window for which message accumulation needs to be re-stated and a second time window for which re-statistics is not required; The message accumulation amount in the message queue at the current statistical moment is calculated based on the historical message accumulation amount of each second time window counted at the previous statistical moment and the message accumulation amount corresponding to each first time window counted newly.

2. The method according to claim 1, characterized in that The method of determining, based on the current statistical moment, the current consumption location, and the expiration time of each time window, a first time window for which message accumulation amount needs to be re-stated and a second time window for which re-statistics is not required from each time window includes: Determine the time interval between the current statistical moment and the previous statistical moment, and the target time window where the current consumption location is located; The time window in which the minimum value of the expiration time is less than or equal to the time interval, the time period between the previous statistical moment and the current statistical moment, and the time period between the current consumption location and the end time of the target time window are all determined as the first time window; The time windows other than the first time window in each time window are determined as the second time window.

3. The method according to claim 1, characterized in that The calculating the message accumulation amount in the message queue at the current statistical moment based on the historical message accumulation amount of each second time window counted at the previous statistical moment and the message accumulation amount corresponding to each first time window counted newly includes: Determine, according to the start time and the end time of each of the first time windows, the messages whose timestamps are within each of the first time windows; Based on the message identifiers of the messages in each of the first time windows, recounting the message accumulation amount in each of the first time windows; the message identifiers at least include a timestamp, a sequence number, and a generating device identifier; The historical message accumulation amount of each second time window and the re-counted message accumulation amount of each first time window are summed to obtain the message accumulation amount in the message queue at the current statistical moment.

4. The method according to claim 3, characterized in that The re-counting of message accumulation amounts in each of the first time windows based on the message identifiers of the messages in each of the first time windows includes: Determine a message identification range of the messages in each of the first time windows based on the message identifications of the messages in each of the first time windows; the message identification range includes a timestamp range, a sequence number range, and a generating device identification range of the messages; According to the generation rule of the message identifiers and the message identifier ranges of the messages in each of the first time windows, the message accumulation amount in each of the first time windows is re-counted.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Upon receiving a new message, insert the new message into the end of the message queue; Determine whether the timestamp of the new message reaches the end time of the current time window; If so, the message accumulation amount in the current time window is counted, and the counted message accumulation amount in the current time window is stored.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: When the first time window for re-counting the message accumulation amount appears, the currently stored historical message accumulation amount of the first time window is updated to the re-counted message accumulation amount.

7. The method according to any one of claims 1 to 4, characterized in that: The obtaining of the current consumption position of the message queue includes: When the user resets the consumption location at the current statistical moment, the reset consumption location is determined as the current consumption location; If the user has not reset the consumption location at the current statistical moment, the time corresponding to the message with the earliest timestamp among the unconsumed messages is determined as the current consumption location.

8. A message accumulation statistics device, characterized in that: include: A message information acquisition module is used to obtain the current consumption position of the message queue and respectively obtain the expiration time corresponding to each time window. The expiration time corresponding to the time window is the expiration time of all messages in the message queue whose timestamps are within the time window. A time window determination module, for determining, from each time window, a first time window for which message accumulation needs to be re-counted and a second time window for which re-counting is not required, based on the current statistical moment, the current consumption location and the expiration time of each time window; The accumulation amount calculation module is used to calculate the message accumulation amount in the message queue at the current statistical moment based on the historical message accumulation amount of each second time window counted at the previous statistical moment and the message accumulation amount corresponding to each first time window counted newly.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images