Message middleware event backlog processing method and apparatus, device, and storage medium

Abstract

The present disclosure relates to a message middleware event backlog processing method, device and equipment and a storage medium. The message middleware event backlog processing method comprises the following steps: reading event messages in a message middleware; the event messages are multiple; performing event cleaning and filtering processing on the event messages according to a preset filtering condition through a handle function; obtaining related data of the event messages of the message middleware; the related data comprises an un-consumed message quantity, a message production speed and a message consumption speed; and controlling a backlog filter according to the related data to perform event backlog filtering processing on the event messages after the event cleaning and filtering processing. The method provided in the present disclosure can improve the event message processing efficiency and timeliness of event message processing.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of Internet, and particularly relates to a message middleware event backlog processing method and device, equipment and a storage medium. BACKGROUND

[0002] In the related art, in the IT (Internet Technology) industry, message middleware is widely applied to various scenes to realize functions such as task queue, decoupling system, broadcast and subscription, data synchronization, and flow control and load balancing. The types of event message sources of the message middleware are various, so the production of the event messages is also difficult to stop or control. The intermediate processing is generally matched and filtered or processed according to the rules of customized development according to existing needs, and then forwarded to the downstream message middleware. If the downstream processing cannot keep up, the messages will accumulate more and more, and the timeliness will be greatly reduced. At present, when event backlog occurs, the existing event messages are analyzed to analyze whether there is redundant data, and then the code for filtering events is added, recompiled and deployed to start processing. However, when all the filterable events are filtered out, there is still event backlog, and there is no other solution except for expansion, and expansion requires time and resources. At this time, the existing technical solution is generally to discard all. This results in low timeliness and low processing efficiency of events. SUMMARY

[0003] The present disclosure provides a message middleware event backlog processing method, device, equipment and storage medium. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a message middleware event backlog processing method is provided, characterized in that it comprises:

[0005] reading event messages in a message middleware; wherein the event messages are a plurality of event messages;

[0006] performing event cleaning and filtering processing on the event messages according to a preset filtering condition through a handle function;

[0007] obtaining related data of the event messages of the message middleware; wherein the related data comprises an un-consumed message quantity, a message production speed and a message consumption speed;

[0008] controlling a backlog filter according to the related data to perform event backlog filtering processing on the event messages after the event cleaning and filtering processing;

[0009] wherein the preset filtering condition and the filtering condition of the backlog filtering processing are set through a self-defined script engine.

[0010] In a possible implementation, after the event cleaning filtering processing of the event messages by the handle function according to the preset filtering condition, the method further includes:

[0011] outputting the event messages after the event cleaning filtering processing to a downstream message middleware;

[0012] The obtaining of the related data of the event messages of the message middleware includes:

[0013] periodically obtaining the number of the unconsumed messages, the message production speed and the message consumption speed of the event messages of the downstream message middleware.

[0014] In a possible implementation, the event backlog filtering processing of the event messages after the event cleaning filtering processing according to the control of the backlog filter includes:

[0015] determining whether the number of the unconsumed messages, the message production speed and the message consumption speed satisfy a backlog filtering condition;

[0016] in a case where the number of the unconsumed messages, the message production speed and the message consumption speed satisfy the backlog filtering condition, controlling the backlog filter to perform the event backlog filtering processing on the event messages after the event cleaning filtering processing.

[0017] In a possible implementation, the backlog filtering condition includes:

[0018] the number of the unconsumed messages is greater than or equal to a first preset number and a message reduction speed is greater than or equal to a preset speed; wherein the message reduction speed is a difference between the message consumption speed and the message production speed.

[0019] In a possible implementation, the backlog filtering condition includes:

[0020] the number of the unconsumed messages is greater than a second preset number, and a product of the message reduction speed and a filtering duration set by the filter is less than the second preset number.

[0021] According to a second aspect of the embodiments of the present disclosure, a message middleware event backlog processing apparatus is provided, including:

[0022] a message reading module, configured to read event messages in a message middleware; wherein the event messages are a plurality of;

[0023] a cleaning filtering module, configured to perform event cleaning filtering processing on the event messages by a handle function according to a preset filtering condition;

[0024] a data acquisition module, configured to acquire relevant data of event messages of the message middleware; wherein the relevant data comprises an unconsumed message quantity, a message production speed and a message consumption speed;

[0025] a backlog filtering module, configured to control a backlog filter to perform event backlog filtering processing on the event messages after the event cleaning and filtering processing;

[0026] wherein the preset filtering condition and the filtering condition of the backlog filtering processing are set by a custom script engine.

[0027] In a possible implementation, the method further includes:

[0028] outputting the event messages after the event cleaning and filtering processing to a downstream message middleware;

[0029] The data acquisition module is configured to:

[0030] periodically acquire the unconsumed message quantity, the message production speed and the message consumption speed of the event messages of the downstream message middleware.

[0031] In a possible implementation, the backlog filtering module includes:

[0032] a determination unit, configured to determine whether the unconsumed message quantity, the message production speed and the message consumption speed satisfy a backlog filtering condition;

[0033] a backlog filtering unit, configured to control the backlog filter to perform event backlog filtering processing on the event messages after the event cleaning and filtering processing in a case where the unconsumed message quantity, the message production speed and the message consumption speed satisfy the backlog filtering condition.

[0034] In a possible implementation, the backlog filtering condition includes:

[0035] the unconsumed message quantity is greater than or equal to a first preset quantity and a message reduction speed is greater than or equal to a preset speed; wherein the message reduction speed is a difference between the message consumption speed and the message production speed.

[0036] In a possible implementation, the backlog filtering condition includes:

[0037] the unconsumed message quantity is greater than a second preset quantity, and a product of the message reduction speed and a filtering duration set by the filter is less than the second preset quantity.

[0038] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, including:

[0039] a processor;

[0040] a memory for storing the processor-executable instructions;

[0041] The processor is configured to execute the instructions to implement the message middleware event backlog processing method according to any one of the first aspect.

[0042] According to a fourth aspect of the embodiments of the present disclosure, a storage medium is provided, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the message middleware event backlog processing method according to any one of the first aspect.

[0043] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising a computer program, when the computer program is executed by a processor, the message middleware event backlog processing method according to any one of the first aspect is implemented.

[0044] The embodiments of the present disclosure provide at least the following beneficial effects:

[0045] In the embodiments of the present disclosure, event messages in a message middleware are read; wherein the event messages are multiple; a handle function is used to perform event cleaning and filtering processing on the event messages according to a preset filtering condition; relevant data of the event messages in the message middleware is obtained; the relevant data includes an un-consumed message quantity, a message production speed and a message consumption speed; and a backlog filter is controlled according to the relevant data to perform event backlog filtering processing on the event messages after the event cleaning and filtering processing. In this way, timely processing of message middleware event backlog can be realized, and the event message processing efficiency and timeliness of event message processing can be improved.

[0046] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings incorporated in the specification and forming a part thereof illustrate embodiments consistent with the present disclosure and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0048] Figure 1 is a flowchart of a message middleware event backlog processing method provided by the embodiments of the present disclosure.

[0049] Figure 2 is an implementation logic diagram of a message middleware event backlog processing method provided by the embodiments of the present disclosure.

[0050] Figure 3 is an implementation logic diagram of a message middleware event backlog processing method provided by the embodiments of the present disclosure.

[0051] Figure 4 is a flowchart of an event cleaning filtering process provided by an embodiment of the present disclosure.

[0052] Figure 5 is a schematic diagram of an event filter provided by an embodiment of the present disclosure.

[0053] Figure 6 is an example diagram of an interface configuration provided by an embodiment of the present disclosure.

[0054] Figure 7 is a flowchart of an event backlog filtering process provided by an embodiment of the present disclosure.

[0055] Figure 8 is a block diagram of a message middleware event backlog processing apparatus provided by an embodiment of the present disclosure.

[0056] Figure 9 is a block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] In order to make the ordinary person skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0058] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following exemplary embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0059] As can be known from the background, in the related art, when event backlog is generated, the existing event messages are analyzed to determine whether there is redundant data, and then the code for filtering events is added, and the system is recompiled, deployed and started for processing. A more mature solution is to let the user set and or condition in the interface, but this solution has limitations on the format of the data, and does not support custom encryption algorithms. When an unsupported encryption and decryption algorithm is encountered, the code needs to be modified and recompiled and deployed. Moreover, the existing technical solutions basically have a problem of filtering or retaining, without considering the priority of the value of the message. When all filterable events are filtered out, and there is still event backlog, there is no other solution except for expansion, which requires time and resources. At this time, the existing technical solutions generally discard all the data and then supplement the necessary data. This results in at least the following technical problems of the existing technical solutions:

[0060] 1. When the filtering rule changes, the update and debugging period is long. When new filtering conditions need to be added or the current filtering rule cannot meet the requirements, the system needs to be recompiled and deployed. Moreover, if a problem occurs during debugging, it is troublesome to modify the code, recompile and deploy. Especially in a secure internal network environment, the process of uploading files is time-consuming and requires a high frequency of updating files during the debugging phase.

[0061] 2. When event backlog is generated, the message cannot be filtered hierarchically, and high-value event messages cannot be prioritized. When all filterable event messages are filtered out, there is still event backlog (which may be caused by changes in the production and consumption speed of event messages in a certain period of time, or may be caused by business adjustment and the need for expansion). In order to ensure the timeliness of events, all backlog data is generally discarded.

[0062] Therefore, the embodiments of the present disclosure provide a message middleware event backlog processing method, device and storage medium to improve the efficiency of event message processing, improve the timeliness of event message processing, improve user value, and achieve cost reduction and efficiency improvement.

[0063] The message middleware event backlog processing method, device and storage medium provided by the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0064] Figure 1 is a flowchart of a message middleware event backlog processing method provided by the embodiments of the present disclosure. The message middleware event backlog processing method can be applied to a server, for example, a single server or a server cluster. As shown in Figure 1 The message middleware event backlog processing method can include the following steps.

[0065] In step S101, event messages in the message middleware are read.

[0066] The event messages are multiple.

[0067] In the message middleware event backlog processing method, the event messages need to be acquired first, which are usually large in quantity. For example, all implementation classes implementing the EventFilter interface can be read by an event processor, and a filter chain List object is formed. The implementation in Java language can be realized by Spring annotation. After the event processor establishes a connection with the message middleware, the event messages of the upstream message middleware can be read. It can be understood that the reading process is similar to the related art, and will not be described here.

[0068] In step S102, the event messages are filtered according to a preset filtering condition by a handle function.

[0069] In the message middleware event backlog processing method, the event messages need to be acquired first, which are usually large in quantity. For example, all implementation classes implementing the EventFilter interface can be read by an event processor, and a filter chain List object is formed. The implementation in Java language can be realized by Spring annotation. After the event processor establishes a connection with the message middleware, the event messages of the upstream message middleware can be read. It can be understood that the reading process is similar to the related art, and will not be described here.

[0070] It should be noted that, due to the large amount of real-time data, the configurations, statistics and other data used by various filter data in redis (Remote Dictionary Server) cache can be cached. All event filters implement the same interface EventFilter. After the event messages pass through the filter, the event messages can be further enriched, supplemented, compressed and filtered. In addition, a name and a priority can be defined for each specific implementation class by customizing a tag.

[0071] In step S103, related data of the event messages in the message middleware is acquired.

[0072] The related data includes the number of unconsumed messages, the message production speed and the message consumption speed.

[0073] In the embodiments of the present disclosure, after the event cleaning filtering processing is performed on the event message, the related data of the event message of the message middleware can be acquired. For example, the number of unprocessed messages, the message production speed and the message consumption speed of the message middleware can be acquired. For example, the unprocessed message condition (e.g., the number of unprocessed messages) of the message middleware and the message production speed and the message consumption speed of a specified topic can be monitored and acquired, wherein the message consumption speed can be the message consumption speed of a specified consumer group (here, mainly taking the kafka message middleware as an example, some message middleware does not distinguish consumers).

[0074] In step S104, the event backlog filter is controlled according to the related data, and the event message after the event cleaning filtering processing is subjected to event backlog filtering processing.

[0075] The preset filtering condition and the filtering condition of the backlog filtering processing are set by a custom script engine.

[0076] In the embodiments of the present disclosure, considering that a large amount of information is subjected to the filtering core cleaning filtering processing, if the consumer downstream of the message middleware cannot consume such a large amount of data, the data needs to be further subjected to the backlog filtering processing to ensure the timeliness of the data. Of course, if the data integrity is required but the data timeliness is not required, the backlog filtering rule is not needed to be used for filtering. For example, after the related data of the event message of the message middleware is acquired, the start and stop of the backlog filter can be controlled according to the number of unprocessed messages, the message production speed and the message consumption speed. After the start of the backlog filter is controlled, the event message after the event cleaning filtering processing can be subjected to the event backlog filtering processing by the backlog filter.

[0077] In the embodiments of the present disclosure, the event message in the message middleware is read, wherein the event message is multiple; the handle function is used to perform the event cleaning filtering processing on the event message according to a preset filtering condition; the related data of the event message of the message middleware is acquired, wherein the related data includes the number of unprocessed messages, the message production speed and the message consumption speed; and the event backlog filter is controlled according to the related data, and the event message after the event cleaning filtering processing is subjected to the event backlog filtering processing. In this way, the timely processing of the message middleware event backlog can be realized, the event message processing efficiency can be improved, and the timeliness of the event message processing can be improved.

[0078] In a possible implementation, the preset filtering condition and the filtering condition of the backlog filtering processing are set by a script engine.

[0079] In the embodiments of the present disclosure, after the mass event messages are obtained, the redundant and repetitive data generally needs to be filtered, screened and compressed, which is a common pre-filtering scheme. Since the content of the event messages is flexible and variable, the filtering conditions are also flexible and variable, therefore, the embodiments of the present disclosure introduce a custom script engine, so that users can directly filter and process data flexibly through scripts. It can be understood that when filtering based on the custom script, the script requires to implement a function handle(e), wherein the input parameter e is a message object, the message object can be processed in the method body, and the return value is a Boolean type, true indicates that the message is retained to the next processing link, and false indicates that the message is discarded. The script uses JavaScript (abbreviated as “JS”) syntax, JDK8 uses nashorn (Nashorn is a lightweight and high-performance JavaScript runtime environment based on JVM), and JDK15+ can use GraalVM. When the demand changes, it can be adjusted immediately, what you see is what you get, without re-deployment and update, so that the event cleaning and filtering can be more flexible, and the continuous compiling and updating operations are avoided, thereby greatly reducing the updating time and adjustment time.

[0080] In a possible implementation, after the event cleaning and filtering processing of the event messages is performed by the handle function according to the preset filtering conditions, the method further includes:

[0081] outputting the event messages after the event cleaning and filtering processing to a downstream message middleware;

[0082] obtaining related data of the event messages of the message middleware, including:

[0083] periodically obtaining the number of unconsumed messages, the message production speed and the message consumption speed of the event messages of the downstream message middleware.

[0084] In the embodiments of the present disclosure, after the event cleaning and filtering processing of the event messages is performed by the handle function according to the preset filtering conditions, that is, after the event processor reads the event messages, for each message, after the data is read from the upstream message middleware in a loop iteration, the data is subjected to the cleaning and filtering processing by the handle method of a series of built-in and custom filter chains, and then output to the downstream message middleware. Then, the number of unconsumed messages, the message production speed and the message consumption speed of the event messages of the downstream message middleware can be periodically obtained. For example, a timing task can be used to monitor the number of unconsumed events, the consumption speed of the event messages and other monitoring information of the event messages of the downstream message middleware.

[0085] In a possible implementation, according to the related data, a backlog filter is controlled to perform event backlog filtering processing on the event messages after the event cleaning and filtering processing, including:

[0086] determine whether the number of unprocessed messages, the message production speed and the message consumption speed satisfy a backlog filtering condition;

[0087] In the case where the number of unprocessed messages, the message production speed and the message consumption speed satisfy the backlog filtering condition, control the backlog filter to perform event backlog filtering processing on the event messages after the event cleaning and filtering processing.

[0088] In an embodiment of the present disclosure, when the event backlog filter is controlled to perform event backlog filtering processing on the event messages after the event cleaning and filtering processing, it can be determined whether the number of unprocessed messages, the message production speed and the message consumption speed satisfy a backlog filtering condition. The backlog filtering condition can include: the number of unprocessed messages is greater than or equal to a first preset number and the message reduction speed is greater than or equal to a preset speed; wherein the message reduction speed is the difference between the message consumption speed and the message production speed. The first preset number can be set according to actual needs, for example, 10000, 15000, etc., and the preset speed can also be set according to actual needs. That is, it can be determined whether the number of unprocessed messages is greater than or equal to the first preset number, and whether the difference between the message consumption speed and the message production speed is greater than or equal to the preset speed. If the number of unprocessed messages is greater than or equal to the first preset number and the message reduction speed is greater than or equal to the preset speed, it can be considered that the current timeliness is poor, and further backlog filtering processing is needed. At this time, the backlog filter can be controlled to perform event backlog filtering processing on the event messages after the event cleaning and filtering processing, and the backlog filter effective state is set to true; otherwise, the backlog filter effective state is set to false.

[0089] In a possible implementation, the backlog filtering condition can further include:

[0090] The number of unprocessed messages is greater than a second preset number, and the product of the message reduction speed and the filtering duration set by the filter is less than the second preset number.

[0091] In the embodiments of the present disclosure, the backlog filtering condition can also be that the number of unconsumed messages is greater than or equal to a second preset number, and the product of the message reduction speed and the filtering duration set by the filter is less than the number of unconsumed messages. The second preset number can be set according to actual needs, for example, 10000, 15000, etc., and the second preset number can be the same as or different from the first preset number. The message reduction speed is the difference between the message consumption speed and the message production speed. That is, it can be determined whether the number of unconsumed messages is greater than the second preset number, and whether the product of the message reduction speed and the filtering duration set by the filter is less than the second preset number. If the number of unconsumed messages is greater than the second preset number, and the product of the message reduction speed and the filtering duration set by the filter is less than the second preset number, it can be considered that the unconsumed messages of the current message middleware cannot be consumed within a specified time, and the timeliness is poor. At this time, the event backlog filtering processing of the event message filtered by the event cleaning filter can be controlled. In this way, the timeliness of the data can be ensured.

[0092] It can be understood that if there is a requirement for data integrity but no requirement for data timeliness, the backlog filtering rule does not need to be used for backlog filtering processing.

[0093] To make the message middleware event backlog processing method provided by the embodiments of the present disclosure clearer, the following will be described in detail in combination with Figures 2-7 As shown in Figure 2 and Figure 3 , the implementation logic of the message middleware event backlog processing method provided by the embodiments of the present disclosure can include the following processing.

[0094] 1. Event cleaning filtering.

[0095] After a large number of event messages come in, it is generally necessary to filter, screen and compress redundant and repetitive data, which is a common pre-filtering scheme. The embodiments of the present disclosure introduce a custom script engine, which makes the cleaning and filtering of events more flexible, and avoids the constant compiling and updating operations, greatly reducing the updating time. The filter core filtering takes effect unconditionally, which filters all event messages that meet the filtering conditions. It is generally used in the case where it is clear and explicit which data needs to be filtered and not processed, and it is necessary to understand the event message situation of the upstream message middleware and which data is needed and which data is not needed by the downstream message middleware, and it is necessary to communicate and confirm with the user and the downstream system manufacturer. The event cleaning filtering adopts a chain architecture, defines a unified interface, and can flexibly add or delete filtering rules according to the configuration. The flow of event cleaning filtering is as shown in Figure 4 , which can include:

[0096] 1) Initialize filter chain

[0097] EventFilter interface, and composes a filter chain List object. The implementation in Java language can be achieved by Spring annotations.

[0098] 2) Read events from the middleware (i.e. the upstream message middleware)

[0099] After the event processor establishes a connection with the message middleware, it can read event messages. This is similar to the conventional technical solution related to the technology, and the code for connecting different message middleware has slight differences, which will not be described in detail.

[0100] 3) Iterate the filter chain and call the handle method

[0101] After the event processor reads the event message, for each event message, the data read from the upstream message middleware is iteratively looped, and the data is filtered by a series of built-in and custom filter chain handle methods, and then output to the downstream message middleware. Due to the large amount of real-time data, redis cache is used to store various filter data such as configuration, statistics, etc. In addition, a scheduled task is needed to monitor the number of unprocessed events, the consumption speed of event messages, and other monitoring information of the downstream message middleware. As shown in Figure 5 All event filters implement the same interface EventFilter. After the event message passes through the filter, it can be further enriched, supplemented, compressed, and filtered. In addition, a custom tag is used to define a name and priority for each specific implementation class.

[0102] The event cleaning and backlog filters provide interface configuration, and create filter instances based on the filter rule data saved in the database. It is worth mentioning that because the content of the event message is flexible and variable, the filtering conditions are also flexible and variable, and the processing of the event message is also difficult to predict. Here, a script engine is directly introduced, allowing users to flexibly filter and process data through scripts, and the demand can be adjusted immediately. What you see is what you get, without the need to redeploy and update, greatly reducing the adjustment time. Figure 6 An example of interface configuration is shown. When filtering based on custom scripts, the script must implement a function handle(e), where the argument e is a message object that can be processed in the method body. The return value is a boolean type, true indicating that the message is retained to the next processing link, and false indicating that the message is discarded. The script uses JavaScript syntax, JDK8 uses the nashorn engine, and JDK15+ can use GraalVM. Custom script specification examples can be as follows:

[0103]

[0104] JDK8 uses nashorn to parse and execute custom JavaScript scripts, parameter passing and return value code example as follows:

[0105]

[0106] JDK15+ uses GraalVM to parse and execute custom JavaScript scripts, parameter passing and return value code example as follows:

[0107]

[0108] 4) Output events to downstream middleware (i.e. downstream message middleware)

[0109] This is no different from the conventional technical solution in the related art, and the code for connecting different message middleware is slightly different, which is not described in detail.

[0110] 2, event backlog filtering.

[0111] After a large amount of information is filtered and cleaned (i.e. cleaning and filtering), if the downstream consumer still cannot consume such a large amount of data, further processing of the data is required to ensure the timeliness of the data. Of course, if there is a requirement for data integrity but no requirement for data timeliness, then the backlog filtering rule does not need to be used for backlog filtering. Similar to the flood warning device of the dam, when the water level rises to a certain position, an alarm will be sent to remind the staff to open the floodgate to discharge water. The principle of event backlog filtering rule is also like this, which allows users to customize different levels and priority filtering rules, and when the event backlog reaches a certain level, the filtering rule can be activated to discard part of the event message to ensure the timeliness of the message. Event backlog filtering has an effective condition, which generally takes effect when the data backlog occurs, and needs to understand the event message of the upstream message middleware and know which event message is more valuable and has higher real-time requirement. The processing of event backlog filtering is the same as that of filter core, only the starting time of the two is different. The specific processing steps are as shown in Figure 7

[0112] 1) Monitor the state of the middleware (i.e. message middleware) at regular intervals. Here, the main task is to start the timer to monitor the un-consumed message of the middleware, as well as the message production speed of the specified topic and the consumption speed of the specified consumer group (here, mainly taking the kafka middleware as an example, some middleware do not distinguish consumers). After obtaining the message production speed and consumption speed, the growth rate of the message can be calculated.

[0113] ​2) The monitor activates the backlog filter in the filter chain in the event handler based on the state. The backlog filter is divided into two categories:

[0114] a) Fixed threshold

[0115] When the number of unprocessed messages reaches a set condition (i.e. greater than or equal to the first preset number) and the message reduction speed (consumption speed - production speed) also reaches a set range (i.e. greater than or equal to the preset speed), the backlog filter activation state is set to true, otherwise the backlog filter activation state is set to false.

[0116] b) Automatic adjustment

[0117] Since the production speed and consumption speed of messages are variable, the automatic condition can allow the backlog filtering rule to make dynamic adjustments according to the situation. The backlog rule activation formula is as follows:

[0118] Unprocessed message number > second preset number (e.g. 10000) and (consumption speed - production speed) * filter set filtering time < second preset number

[0119] Unprocessed message number > 10000 The 10000 in 10000 is configured through a configuration file, mainly to avoid starting the backlog filtering rule when the message just arrives and has not been consumed yet during the automatic adjustment of the backlog rule. (Consumption speed - production speed) * filter set time < 10000, which calculates through the formula whether the unprocessed messages in the current message middleware can be consumed within the specified time.

[0120] In this way, the embodiments of the present disclosure can dynamically adjust the data filtering and cleaning processing of different formats and different data sources by introducing a suitable and efficient script engine to process events of various data sources and data formats, and can efficiently and flexibly cope with different types of message processing and cleaning, thereby greatly improving the flexibility of the filtering rules. Meanwhile, the event backlog filtering with the nature of a fuse and flood warning is introduced, and whether the data of different priority messages is filtered is determined by monitoring the backlog of the message middleware and comparing the production and consumption rates, which can be used as an emergency plan to ensure the timeliness of high-value events. When the event backlog occurs, the processing of high-value events is guaranteed first. In this way, compared with the existing event filtering scheme, the event cleaning processing problem of being unable to adapt to various data formats, the long update and debugging period, the low efficiency, and the problem of discarding all events in a one-size-fits-all manner when the event backlog occurs are solved. The embodiments of the present disclosure comprehensively use the prior multistage filtering means, introduce the script engine scheme, and design the method provided by the embodiments of the present disclosure by referring to the processing strategies of the fuse and flood warning. The method provided by the embodiments of the present disclosure can efficiently and flexibly process event messages of various data structures, improve the timeliness of events, improve the user experience, reduce the cost and skill requirements of the operation personnel, effectively realize the commercial value realization and cost reduction and efficiency improvement.

[0121] Based on the same inventive concept, the embodiments of the present disclosure also provide a message middleware event backlog processing apparatus, as shown in Figure 8 Figure 8 is a block diagram of a message middleware event backlog processing apparatus according to an example embodiment. Referring to Figure 8 , the message middleware event backlog processing apparatus 800 can include:

[0122] a message reading module 810, configured to read event messages in a message middleware; wherein the event messages are a plurality of event messages;

[0123] a cleaning and filtering module 820, configured to perform event cleaning and filtering processing on the event messages according to a preset filtering condition through a handle function;

[0124] a data acquisition module 830, configured to acquire related data of the event messages in the message middleware; wherein the related data includes the number of unconsumed messages, the message production speed, and the message consumption speed;

[0125] a backlog filtering module 840, configured to control a backlog filter according to the related data to perform event backlog filtering processing on the event messages after the event cleaning and filtering processing;

[0126] wherein the preset filtering condition and the filtering condition of the backlog filtering processing are set through a self-defined script engine.

[0127] ​In a possible implementation, the method further includes:

[0128] outputting the event messages filtered by the event cleaning and filtering to a downstream message middleware;

[0129] The data acquisition module is configured to:

[0130] periodically acquire the number of unprocessed messages, the message production speed, and the message consumption speed of the event messages of the downstream message middleware.

[0131] In a possible implementation, the backlog filtering module includes:

[0132] a determination unit configured to determine whether the number of unprocessed messages, the message production speed, and the message consumption speed satisfy a backlog filtering condition;

[0133] a backlog filtering unit configured to control a backlog filter to perform event backlog filtering on the event messages filtered by the event cleaning and filtering, in a case where the number of unprocessed messages, the message production speed, and the message consumption speed satisfy the backlog filtering condition.

[0134] In a possible implementation, the backlog filtering condition includes:

[0135] the number of unprocessed messages is greater than or equal to a first preset number, and a message reduction speed is greater than or equal to a preset speed, where the message reduction speed is a difference between the message consumption speed and the message production speed.

[0136] In a possible implementation, the backlog filtering condition includes:

[0137] the number of unprocessed messages is greater than a second preset number, and a product of the message reduction speed and a filtering duration set by the filter is less than the second preset number.

[0138] As to the apparatus in the above-mentioned embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described here in detail.

[0139] According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0140] Figure 9A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0141] like Figure 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 may also store various programs and data required for the operation of the device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0142] Multiple components in electronic device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows electronic device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0143] The computing unit 901 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs various methods and processes described above, such as the message middleware event backlog processing method. For example, in some embodiments, the message middleware event backlog processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded onto the RAM 903 and executed by the computing unit 901, one or more steps of the message middleware event backlog processing method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the message middleware event backlog processing method by any other suitable means, such as by means of firmware.

[0144] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0145] Program code of a computer program product for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, implements the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0146] In the context of the present disclosure, a storage medium can be a tangible medium which can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A storage medium can be a machine-readable signal medium or a machine-readable storage medium. Storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of storage media can include one or more lines of electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0147] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0148] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0149] The computer system can include clients and servers. This relationship can be. remote, where each server is stored on a remote computer from a client. The clients and the servers can be connected through a communication network. The relationship can be a client-server relationship over a network. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS"). The server can also be a server of a distributed system or a server combined with a blockchain.

[0150] It should be understood that the steps shown above can be reordered, added to, or deleted from. For example, the steps described in the present disclosure can be executed in parallel, in sequence, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0151] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for handling backlogged events in a message middleware, characterized in that, include: Read event messages from the message middleware; wherein, there are multiple event messages; The event messages are cleaned and filtered according to preset filtering conditions using the handle function. Obtain relevant data about the event messages of the message middleware; wherein, the relevant data includes the number of unconsumed messages, message production rate, and message consumption rate; The backlog filter is controlled based on the relevant data to perform backlog filtering on the event messages after event cleaning and filtering. The preset filtering conditions and the backlog filtering conditions are set through a custom script engine.

2. The message middleware event backlog processing method according to claim 1, characterized in that, After performing event cleaning and filtering processing on the event messages according to preset filtering conditions using the handle function, the process further includes: Output the event messages after event cleaning and filtering to the downstream message middleware; The process of obtaining relevant data for the event messages of the message middleware includes: The number of unconsumed messages, the message production rate, and the message consumption rate of the downstream message middleware are periodically obtained.

3. The message middleware event backlog processing method according to claim 2, characterized in that, The step of controlling the backlog filter based on the relevant data to perform event backlog filtering on the event messages after event cleaning and filtering includes: Determine whether the number of unconsumed messages, the message production rate, and the message consumption rate meet the backlog filtering conditions; If the number of unconsumed messages, the message production rate, and the message consumption rate meet the backlog filtering conditions, the backlog filter is controlled to perform backlog filtering on the event messages after event cleaning and filtering.

4. The message middleware event backlog processing method according to claim 3, characterized in that, The backlog filtration conditions include: The number of unconsumed messages is greater than or equal to a first preset number and the message reduction rate is greater than or equal to a preset rate; wherein, the message reduction rate is the difference between the message consumption rate and the message production rate.

5. The message middleware event backlog processing method according to claim 3, characterized in that, The backlog filtration conditions include: The number of unconsumed messages is greater than the second preset number, and the product of the message reduction rate and the filter duration set by the filter is less than the second preset number.

6. A message middleware event backlog processing device, characterized in that, include: The message reading module is used to read event messages from the message middleware; wherein, there are multiple event messages. The cleaning and filtering module is used to perform event cleaning and filtering processing on the event messages according to preset filtering conditions through the handle function. The data acquisition module is used to acquire relevant data of the event messages of the message middleware; wherein, the relevant data includes the number of unconsumed messages, message production speed, and message consumption speed; The backlog filtering module is used to control the backlog filter according to the relevant data, and to perform backlog filtering on the event messages after the event cleaning and filtering process. The preset filtering conditions and the backlog filtering conditions are set through a custom script engine.

7. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the message middleware event backlog processing method as described in any one of claims 1 to 5.

8. A storage medium, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the message middleware event backlog processing method as described in any one of claims 1 to 5.

9. A computer program product comprising a computer program that, when executed by a processor, implements the message middleware event backlog processing method as described in any one of claims 1 to 5.

Images