A message processing method, apparatus and electronic device

Abstract

The application provides a message processing method and device and electronic equipment to avoid the problem that the performance of a relational database is reduced due to message concentration and the system is stuck or even crashes due to memory soaring when the message is currently stored in the relational database. The method comprises the following steps: in response to obtaining a message, determining type information of the message; wherein the type information comprises a correlation between the message and a business scenario of the message; for the messages with the same type information, grouping the messages based on a timestamp carried by the messages to obtain a plurality of message groups; wherein the timestamps of the messages in the message groups are located in the same preset time period; and caching the message groups to a to-be-processed queue.

Description

Technical Field

[0001] This application relates to the field of message processing technology, and in particular to a message processing method, apparatus and electronic device. Background Technology

[0002] Currently, when processing messages, systems often use message middleware (such as Kafka) to categorize messages, thereby decoupling message production and consumption. In the aforementioned message middleware, after messages are categorized and consumed by topic, they are stored in a relational database. Then, message polling and computation are performed on the relational database to retrieve grouped messages.

[0003] However, in some business scenarios, messages are characterized by high reporting frequency and large volume. When messages are generated intensively, storing grouped messages in a relational database using the aforementioned method will cause the relational database to face excessive storage pressure in a short period of time. This can easily lead to performance issues such as data read / write anomalies and reduced query efficiency, including data loss. In this case, the aforementioned system also needs to connect to the relational database more frequently, resulting in performance degradation and excessive memory consumption. Furthermore, the large amount of data resources generated to query complete grouped messages also consumes a significant amount of system memory, leading to memory spikes. Summary of the Invention

[0004] This application provides a message processing method, apparatus, and electronic device to avoid the problems that occur when messages are persistently stored in relational databases, such as performance degradation due to message density and system lag or even crashes due to memory spikes.

[0005] Firstly, embodiments of this application provide a message processing method, including:

[0006] In response to receiving a message, the type information of the message is determined; wherein, the type information includes the relevance between the message and the business scenario of the message;

[0007] For messages with the same type of information, the messages are grouped based on the timestamps they carry to obtain multiple message groups; wherein the timestamps of the messages in the message group are within the same preset time period;

[0008] The message group is cached in the queue to be processed.

[0009] One possible implementation involves grouping messages with the same type of information into multiple message groups based on the timestamps they carry, including:

[0010] Read the timestamp from the message; wherein the timestamp corresponds to the time the message was generated;

[0011] Among the messages with the same type information, the message corresponding to the earliest target timestamp is determined to be the reference message;

[0012] The moment when the reference message is obtained is determined as the first moment, and the second moment is determined based on the first moment and the obtained longest waiting time threshold;

[0013] The message group is determined based on a first time window; wherein the start time of the first time window is the first time and the end time is the second time.

[0014] One possible implementation, wherein determining the message group based on a first time window, includes:

[0015] The messages whose arrival time falls within the first time window are identified as the intermediate message group;

[0016] For the intermediate message group, a third moment is determined based on the target timestamp and the preset interval of the acquired messages;

[0017] In the intermediate message group, messages whose timestamps are located between the target timestamp and the third time point are identified as a group of messages.

[0018] One possible implementation includes determining the message type information, which includes:

[0019] In the message, a first object is identified based on a first target key; wherein the first object includes the first target key and a first target value, and the first target value indicates the information of the business scenario and the relevance.

[0020] In a preset first mapping relationship, the type information corresponding to the first target value is queried; wherein, the first mapping relationship includes the correspondence between the first field and the type information.

[0021] One possible implementation includes determining the message type information, which includes:

[0022] In the message, the first object and the second object are identified based on the first target key and the second target key; wherein, the second object includes the second target key and the second target value, and the second target value indicates the business rule corresponding to the business scenario;

[0023] In the preset second mapping relationship, query the type information corresponding to the first target value and the second target value.

[0024] One possible implementation, before grouping the messages based on the timestamps carried by the messages with the same type information, further includes:

[0025] Determine the target thread pool corresponding to the message;

[0026] Using threads in the target thread pool, the message is added to the first queue, and a dequeue rule is determined for the message in the first queue, so that the message is dequeued from the first queue according to the dequeue rule and enters the message queue to be grouped.

[0027] Then, for messages with the same type information, the messages are grouped based on the timestamps they carry, resulting in multiple message groups, including:

[0028] For messages with the same type information, the messages in the message queue to be grouped are grouped based on the timestamp carried by the message to obtain the multiple message groups.

[0029] One possible implementation is that the message is dequeued from the first queue using a blocking take.

[0030] One possible implementation, after caching the message group to the processing queue, further includes:

[0031] In response to the fact that all messages in the message group in the pending queue are in a completed state, process the messages in the message group; or,

[0032] In response to a situation where at least one message in the message group in the pending queue is not in the completed state, a preset time is waited to determine the state of the messages in the message group until all messages in the message group are in the completed state, and then the messages in the message group are processed.

[0033] Secondly, embodiments of this application provide a message processing apparatus, including:

[0034] A type unit is used to determine the type information of a message in response to receiving a message; wherein the type information includes the relevance between the message and the business scenario of the message;

[0035] A grouping unit is used to group messages with the same type information into multiple message groups based on the timestamps carried by the messages; wherein the timestamps of the messages in the message group are within the same preset time period;

[0036] A caching unit is used to cache the message group to a queue to be processed.

[0037] In one possible implementation, the grouping unit is specifically used to read the timestamp in the message; wherein the timestamp corresponds to the generation time of the message; among the messages with the same type information, the message corresponding to the earliest target timestamp is determined as a reference message; the time when the reference message is obtained is determined as a first time, and a second time is determined based on the first time and the obtained longest waiting time threshold; the message group is determined based on a first time window; wherein the start time of the first time window is the first time, and the end time is the second time.

[0038] In one possible implementation, the grouping unit is specifically used to determine messages whose acquisition time falls within the first time window as an intermediate message group; for the intermediate message group, a third time is determined based on the target timestamp and the preset interval of the acquired messages; in the intermediate message group, messages whose timestamps fall between the target timestamp and the third time are determined as a group of messages.

[0039] In one possible implementation, the type unit is specifically used to identify a first object in the message based on a first target key; wherein the first object includes the first target key and a first target value, the first target value indicating information about the business scenario and the relevance; and to query the type information corresponding to the first target value in a preset first mapping relationship; wherein the first mapping relationship includes the correspondence between the first field and the type information.

[0040] In one possible implementation, the type unit is further configured to identify the first object and the second object in the message based on a first target key and a second target key; wherein the second object includes the second target key and a second target value, the second target value indicating a business rule corresponding to the business scenario; and in a preset second mapping relationship, query type information corresponding to the first target value and the second target value.

[0041] In one possible implementation, the device further includes a thread unit, configured to determine the target thread pool corresponding to the message; use threads in the target thread pool to add the message to a first queue, and determine a dequeue rule for the message in the first queue, so that the message is dequeued from the first queue based on the dequeue rule and enters the message queue to be grouped;

[0042] The grouping unit is further configured to group messages in the message queue to be grouped based on the timestamp carried by the messages, for messages with the same type information, to obtain the multiple message groups.

[0043] One possible implementation is that the message is taken in a blocking manner when it is dequeued from the first queue.

[0044] In one possible implementation, the device further includes a status unit, configured to process messages in the message group in response to all messages in the message group in the pending queue being in the completed state; or, in response to at least one message in the message group in the pending queue being in a state other than the completed state, wait a preset time to determine the state of the messages in the message group until all messages in the message group are in the completed state, and then process the messages in the message group.

[0045] Thirdly, embodiments of this application provide an electronic device, including:

[0046] Memory, used to store program instructions;

[0047] A processor is configured to invoke program instructions stored in the memory and execute the steps included in the method of the first aspect and any possible implementation according to the obtained program instructions.

[0048] Fourthly, embodiments of this application provide a computer storage medium on which a computer program is stored.

[0049] When the computer program is executed by a processor, it implements the steps of the method described in the first aspect and any possible implementation.

[0050] Fifthly, embodiments of this application provide a computer program product, including a computer program.

[0051] When the computer program is executed by a processor, it implements the steps of the method described in the first aspect and any possible implementation.

[0052] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects:

[0053] In the message processing method provided in this application embodiment, multiple message groups are obtained by grouping messages with the same type information according to the timestamps carried by the messages. Since the messages are generated at the same time or successively by one or more message sources used to monitor events, the aforementioned message groups obtained by timestamp grouping can actually group the messages of various events into one group. Therefore, when caching the queue to be processed in units of message groups, on the one hand, it can accurately obtain the message groups that reflect the events monitored by the message sources, while avoiding the problem of insufficient performance of relational databases due to large read and write operations when persisting messages in relational databases, and the problem of system performance degradation, excessive memory consumption, and system lag and crash due to frequent connection to relational databases required by frequent message reporting to the system.

[0054] On the other hand, it avoids the problem of being unable to achieve fine-grained grouping when using topics to classify messages due to the single dimension of topic classification. Moreover, the messages in the queue to be processed are actually stored and retrieved in units of message groups, and the number of messages in a message group is far less than the number of messages in a topic. Therefore, the message groups in the queue to be processed have the advantages of high storage and retrieval efficiency and good timeliness, avoiding the situation of message blockage after receiving messages. This allows for the real-time processing of a large number of dense messages, ensuring that the system's cache space can accommodate the messages in the queue to be processed, achieving the purpose of caching the queue to be processed, and avoiding the situation where the cache cannot support it and can only choose the relational database for persistent storage.

[0055] Furthermore, using the aforementioned message group as the processing unit of the pending queue can ensure that the information in the message group is processed in a timely manner. Correspondingly, the events corresponding to the message group can also be processed in a timely manner, thus ensuring the timeliness of events such as power outage repairs at a substation or fixed line.

[0056] Furthermore, by pre-setting thread pools for different types of messages, the target thread pool corresponding to the message can be determined based on the message type information when a message is received, further improving message processing efficiency. Especially in scenarios with multiple message sources and complex message type information generated by each source, the message processing method provided in this application embodiment exhibits significantly smoother message processing characteristics.

[0057] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The purposes and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit this disclosure. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0059] Figure 1 This application provides a flowchart illustrating a message processing method.

[0060] Figure 2 A flowchart illustrating a method for placing messages into a message queue to be grouped, provided as an embodiment of this application;

[0061] Figure 3 This is a schematic diagram of a message processing method provided in an embodiment of this application;

[0062] Figure 4 This is a schematic diagram of the structure of a message processing device provided in an embodiment of this application;

[0063] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0064] In response to the current issues of high read / write pressure and excessive system memory consumption in persistent message storage databases used in scenarios with intensive message reporting, this application provides a message processing method: after grouping messages according to the timestamps they carry, the message groups are cached to avoid storing message groups in relational databases, which have insufficient performance and consume excessive system memory.

[0065] Furthermore, this method also improves the performance of the information processing system by caching messages in the form of a queue and taking advantage of the inbound and outbound characteristics of Java queues to dynamically adjust and process message groups in the queue in real time.

[0066] Furthermore, since message consumers often consume messages in groups, this solution groups messages more finely based on their timestamps, reducing the number of messages in each group and lowering the pressure on the message processing system to process messages in the queue.

[0067] The Java queue, or Queen, described above, is a special type of linear list that follows the basic principles of First-In-First-Out (FIFO) and Last-In-Last-Out (LIFO). The Queue interface is defined in the Java Collections Framework and represents a FIFO queue. The Queue interface inherits from the Collection interface and defines a set of methods for manipulating the elements in the queue.

[0068] To better understand the above technical solutions, the technical solutions of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this application and the specific features in the embodiments are detailed descriptions of the technical solutions of this application, rather than limitations on the technical solutions of this application. In the absence of conflict, the embodiments of this application and the technical features in the embodiments can be combined with each other.

[0069] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.

[0070] Please refer to Figure 1 This application proposes a message processing method to avoid the problems of storing message groups in a relational database, which has insufficient performance and consumes too much system memory. This method can be used in message processing systems.

[0071] The method specifically includes the following implementation steps:

[0072] Step 101: In response to receiving a message, determine the type information of the message.

[0073] The type information includes the relevance between messages and their respective business scenarios.

[0074] Specifically, a message includes at least the identifier of the message source that sent the message and a timestamp corresponding to the time the message was generated or determined.

[0075] The message can be generated by different message sources at different or the same time, or it can be generated by the same message source at different times.

[0076] In one possible implementation, the message can be proactively reported by the message source so that the message processing system can obtain the message.

[0077] Alternatively, in one possible implementation, the message processing system can periodically send instructions to the message source to query whether the message source contains a message. When the message source contains a message, the message will be obtained by the message processing system.

[0078] Furthermore, the aforementioned type information is actually a refined classification of messages based on their business context. In some embodiments, type information may include the business context of the message and the relevance between the message and its business context. Here, relevance can be equated to the importance of the message in its business context; a higher relevance means that the message is more important to the business context.

[0079] Accordingly, type information can be divided into a preset number of levels based on relevance, i.e., the aforementioned importance, such as level one, level two, level three, etc.

[0080] For example, in a business scenario involving anomaly monitoring of substation A, the type information could include anomaly monitoring of substation A and a first-level message. This first-level message indicates that the message has the highest relevance to the anomaly monitoring event of substation A.

[0081] Alternatively, in some embodiments, the type information can be further subdivided into business scenarios to obtain more granular types. Taking the anomaly monitoring of substation A as an example, the business scenario is defined as the lines in substation A. If substation A supplies power to lines α, β, and γ, the type information can include anomalies in line α, and the correlation between the anomaly and line α; anomalies in line β, and the correlation between the anomaly and line β; and anomalies in line γ, and the correlation between the anomaly and line γ.

[0082] The following provides a detailed explanation of how to determine type information:

[0083] Since the message can be in JSON format, and JSON objects include key-value pairs, we can first identify the first object in the message based on the first target key. The first object includes the first target key and the first target value. This first target value indicates information about the business scenario and the aforementioned relevance.

[0084] Then, the type information corresponding to the first target value can be queried from the preset first mapping relationship. The first mapping relationship includes the correspondence between the first target value and the type information.

[0085] Thus, in this embodiment, the first target key is an identifier corresponding to the business scenario and the message, while the first target value is an identifier of the specific business scenario and relevance. For example, if {"business scenario": "001"}, then the object is determined to be the first object based on the first target key "business scenario", so "001" is the first target value. Therefore, the type information corresponding to "001" can be determined by querying the first mapping relationship.

[0086] Furthermore, if a second target key can be identified in the message, the object containing that second target key can be determined as a second object. This second object includes a second target key and a second target value, whereby the second target value can be used to indicate business rules corresponding to the business scenario.

[0087] Then, in the preset second mapping relationship, the type information corresponding to the first target value and the second target value is queried. That is, the second mapping relationship includes the correspondence between the first target value, the second target value, and the type information. For example, if the second object is {rule: 001}, then the second target key rule indicates that the corresponding second target value is a specific business rule. By reading and querying rule, the corresponding second target value 001 can be determined as the identifier of the corresponding business rule. Thus, based on business rule 001 (i.e., the second target value) and the first target value corresponding to the aforementioned business scenario, the type information of the current message can be queried in the second mapping relationship.

[0088] The mapping relationships between the first target value, the second target value, and the type information in this second mapping relationship can be seen in the table below:

[0089]

[0090] Note: The letters a, b, c, A, and B in the table only indicate that the representation of each target value and type information is different. The specific representation logic or method may be the same or different.

[0091] As mentioned earlier, messages can be generated by different message sources at different or the same times, or they can be generated by the same message source at different times. Therefore, the above business rules can instruct the calculation logic for each message, and this calculation logic corresponds to the business scenario.

[0092] The following are examples illustrating business rules:

[0093] Taking the aforementioned business scenario of anomaly monitoring of substation A lines α and β as an example, assuming that equipment 1 and equipment 2 are deployed on line α as message sources, and equipment 3 is deployed on line β as a message source, the business rule could be to process messages from equipment 2 and equipment 3 first, and then process messages from equipment 1. Under this business rule, even if a message from equipment 1 is obtained first, it is still added to the first queue. This continues until messages from equipment 2 and equipment 3 enter the first queue. Based on this business rule, it is determined that messages from equipment 1 to equipment 3 can participate in grouping, so messages from equipment 2 and equipment 3, along with the message from equipment 1, are added to the message queue to be grouped.

[0094] As can be seen from the example above, the aforementioned business rules can be used to determine when a message is delayed from being dequeued from the first queue. However, to determine when a message enters the message queue from the first queue, it is also necessary to determine the supported delay time for the message based on the message type information, or the business rules within the type information. This is to avoid situations where, in the aforementioned example, messages from device 2 and / or device 3 are delayed due to network or other special factors, preventing messages from device 1 from being dequeued from the first queue.

[0095] Continuing with the example: Since the information types of messages generated by each message source can be multiple, suppose device 1 and device 2 are deployed on line α as message sources, and each message source generates both type A and type B messages. If the business rule is to process type A messages from each device first, then process type B messages, in some cases, type B messages from device 1 are actually received first.

[0096] According to the business rules, the allowed delay time for this type B message is determined to be 60 seconds. The type B message will wait in the first queue for 60 seconds before being dequeued, so that if a type A message is obtained within those 60 seconds, the type A message can be dequeued before the type B message. If a type A message for device 1 is not obtained within 60 seconds, the type B message will still be dequeued.

[0097] If a message of type A is obtained within 60 seconds after a message of type B enters the first queue, the message of type A can be dequeued from the first queue as soon as it enters, while the message of type B continues to wait until the 60th second before being dequeued. Alternatively, the message of type B can be dequeued immediately after the message of type A, without waiting until the 60th second.

[0098] To avoid performance degradation caused by periodic queries, in some embodiments, a blocking take is used when messages are dequeued from the first queue.

[0099] Furthermore, to avoid situations where threads are occupied by previously acquired messages when a message containing core information is acquired in a message-intensive reporting scenario, this embodiment pre-sets corresponding thread pools for different types of information. This ensures that when messages of each type of information are acquired, especially highly relevant core messages, there are threads available for corresponding processing. The threads in this thread pool are mainly used to add messages of the corresponding type of information to the first queue, and to dequeue messages from the first queue and add them to the message queue to be grouped according to the dequeue rules.

[0100] Furthermore, the number of threads in the corresponding thread pool can be dynamically adjusted based on the generation frequency and other characteristics of different types of messages.

[0101] In some embodiments, corresponding to the aforementioned pre-set thread pool, after determining the message type information, the message can be stored and retrieved using threads in the target thread pool. Please refer to [reference needed] for details. Figure 2 .

[0102] Step 201: Determine the target thread pool corresponding to the message.

[0103] Specifically, the target thread pool corresponding to the message can be determined based on a preset third mapping relationship. This third mapping relationship includes the correspondence between type information and target thread pools.

[0104] In some embodiments, the target thread pool can be determined directly after the type information is determined.

[0105] Alternatively, in some embodiments, because some messages are time-sensitive and require immediate processing or response, such as when the message can directly determine the specific situation of the event monitored by the message source, or when the event monitored by the message is an event that requires urgent processing, such time-sensitive messages can be processed directly without performing subsequent steps, i.e., grouping is no longer required.

[0106] Therefore, before executing step 201, it can be determined whether the message needs to be grouped; if yes, then step 201 is executed; if no, then the message is processed directly, or the message is added to the message queue described in step 103.

[0107] When determining whether a message needs to be grouped, a third object can be identified through the third target key in the message. The grouping requirement is then determined based on the third target value within that third object. For example, if the third object is {"Does it need to be grouped?": "Yes"}, then the third target key is "Does it need to be grouped?", and the third target value is "Yes".

[0108] Alternatively, the required message grouping can be determined by the allowed delay time. If the delay time is less than a preset delay threshold, then no grouping is required; if the delay time is greater than or equal to the preset delay threshold, then grouping is required.

[0109] Step 202: Using threads in the target thread pool, add messages to the first queue, and determine the dequeue order for messages in the first queue according to the delay time supported by the messages, so that messages are dequeued from the first queue according to the dequeue rule and enter the message queue to be grouped.

[0110] Specifically, the threads in the target thread pool are used to add messages to the first queue and add messages from the first queue to the message queue to be grouped according to the dequeue rules.

[0111] In some embodiments, the dequeue order in the first queue can be determined directly based on the delay time corresponding to the type information (or the business rules in the type information). This dequeue rule is based on the message's delay time. That is, messages with shorter allowed delay times are dequeued first, and messages with longer allowed delay times are dequeued later. For messages with the same allowed delay time or a time difference less than the corresponding threshold, the message with the earlier timestamp can be dequeued first.

[0112] Since the monitoring of some events requires the joint completion of multiple message sources, the message processing system also needs to jointly analyze and calculate the aforementioned multiple message sources to obtain information about the corresponding events. Therefore, in some embodiments, the dequeue rules can be determined based on the aforementioned business rules and the allowed delay time of the messages.

[0113] This dequeue rule is used to indicate the rules for determining the dequeue order of messages in the first queue.

[0114] Specifically, the dequeue rule is to determine, within the allowed delay time of the first message, the second message waiting to be added to the first queue according to the business rules corresponding to the first message, and then wait. The second message can be one or multiple messages.

[0115] When the second message that the first message is waiting for is added to the first queue within the allowed delay time of the first message, the dequeue order of the first message and the second message is determined according to the business rules, and the first message and the second message are dequeued.

[0116] If the allowed delay time for a message is exceeded, the message will no longer wait. Based on the business rules corresponding to the first message, the message that has been added to the first queue will be determined, and the dequeue order of the first message and the messages that the first message has been waiting for will be determined, and each message will be dequeued from the first queue.

[0117] Taking the aforementioned business scenario of anomaly monitoring of substation A lines α and β as an example, with device 1 and device 2 deployed on line α as message sources, and device 3 deployed on line β as a message source, the business rule is to process messages from device 2 and device 3 first, and then process messages from device 1. For example: After receiving a message from device 1, after determining that the allowed delay time for device 1 is 10 seconds, the message from device 1 is added to the first queue. If messages from device 2 and device 3 are added to the first queue within the allowed delay time, then the messages from device 2 and device 3, and then the message from device 1, are dequeued sequentially and added to the message queue to be grouped.

[0118] If no message from device 2 or device 3 is added to the first queue within the allowed delay time, then the message from device 2 or device 3 that was added to the first queue and the message from device 1 are dequeued from the first queue in sequence and added to the waiting queue in sequence.

[0119] Alternatively, if neither device 2 nor device 3's messages are added to the first queue within the allowed delay time, then device 1's message is dequeued from the first queue and added to the waiting-to-be-grouped queue.

[0120] Step 102: For messages with the same type information, group the messages based on the timestamps carried by the messages to obtain multiple message groups.

[0121] In this context, the timestamps of messages within a message group are of the same time length.

[0122] In some embodiments, the timestamp is actually the moment when the message source generates (i.e. determines) the message.

[0123] In some embodiments, since a message source typically monitors events in its environment based on stored or received instructions, it determines a message corresponding to the event when the event occurs. Therefore, the message generation step is triggered by the monitored event. Thus, the timestamp can also be the moment the event monitored by the message source occurs.

[0124] The message source can be a monitoring system or device, and the events it monitors can be the status of the monitored object. For example, when the monitored object exhibits an abnormal state, the message source determines that the monitored event has occurred, uses the time of the event as the timestamp in its determined message, and records the corresponding event information in the message.

[0125] Furthermore, messages can be cached in a non-relational database, such as Redis (Remote Dictionary Server). In Redis, messages are stored in key-value pairs. Therefore, in this embodiment, after messages are cached in Redis, Redis includes a set of type information (keys) and a set of messages (values).

[0126] When grouping messages, the scheduler can periodically retrieve messages from the relational database by key and sort them according to the time of retrieval or the order of timestamps in the messages.

[0127] Furthermore, to avoid situations in step 101 where, due to network latency in the message processing system or congestion in the message source's message queue, messages that should have been retrieved earlier (i.e., those with earlier timestamps) are not retrieved, while messages with later timestamps are retrieved first, the message retrieved at 9:10:21 may have a timestamp of 9:10:20, the message retrieved at 9:10:26 may have a timestamp of 9:10:25, and the message retrieved at 9:10:31 may have a timestamp of 9:10:21. To avoid missing the message retrieved at 9:10:31 with a timestamp of 9:10:21, thus preventing insufficient analysis of the event information corresponding to that message, a longest waiting time threshold can be used to determine message grouping. Specifically, the timestamp in the message can be read first. The timestamp corresponds to the message's generation time.

[0128] Then, among messages with the same type information, the message corresponding to the earliest target timestamp is identified as the reference message.

[0129] Next, the moment when the reference message is obtained is determined as the first moment, and the second moment is determined based on the first moment and the longest waiting time threshold. In other words, the first moment is added to the longest waiting time threshold to obtain the second moment.

[0130] Finally, the first moment is taken as the start moment and the second moment as the end moment to obtain the first time window. The message group is then determined based on this first time window.

[0131] In some embodiments, messages within a first time window can be identified as a message group. After each message group is identified, the reference message and target timestamp are redefined to determine the next message group, thus obtaining multiple message groups.

[0132] Alternatively, to reduce the number of messages in a message group and make the message group division more precise, messages whose acquisition time falls within the first time window can be identified as intermediate message groups.

[0133] Then, for the intermediate message group, the third moment is determined based on the target timestamp and the obtained message preset interval. That is, the target timestamp is added to the message preset interval to obtain the third moment. Here, the message preset interval indicates the time difference between the timestamps of messages corresponding to the same event, and this time difference can be a preset empirical value.

[0134] Thus, within the aforementioned intermediate message group, messages with timestamps between the target timestamp and the third time point can be identified as a single message group. Therefore, the intermediate message group may comprise one or more message groups.

[0135] In the above embodiment, the first time window ensures that all messages to be collected are obtained, and then the messages corresponding to the same event are divided into a group of messages based on the timestamp, the target timestamp and the third time.

[0136] For example, with a preset message interval of 5 seconds, the timestamp of the message obtained at 9:10:21 is 9:10:20, the timestamp of the message obtained at 9:10:26 is 9:10:25, and the timestamp of the message obtained at 9:10:31 is 9:10:21. In this scenario, the error of incorrectly grouping the message with the timestamp of 9:10:25 obtained at 9:10:26 with the other two messages into one message group can be avoided.

[0137] Furthermore, in the embodiment where messages are added sequentially to the first queue and the message queue to be grouped using the aforementioned pre-set thread pool, messages in the message queue to be grouped can be grouped during the grouping process. The specific grouping method is the same as the aforementioned method of determining message groups based on the first time window and / or the target timestamp and the third moment, and will not be repeated here.

[0138] Step 103: Cache multiple message groups into the pending queue.

[0139] Specifically, the processing of messages in the queue to be processed is asynchronous with the aforementioned steps 101-102. This is because of the queue characteristics of the queue to be processed, namely, the extraction (processing) of messages in the queue to be processed and the storage of messages (i.e., adding message groups to the queue to be processed) do not affect each other, and the determination of the aforementioned message groups is also asynchronous with the processing of messages in the queue to be processed.

[0140] In other words, while messages are being generated in a dense manner, and while step 101 is acquiring a continuous stream of messages and adding them to the queue to be processed, the message processing system processes the message groups in the queue to be processed in order of their entry into the queue.

[0141] The aforementioned processing of messages in the queue refers to the calculation and analysis of events monitored by the message source based on the message and the corresponding summoning information. This summoning information is obtained by the message processing system after receiving the message, simultaneously executing steps 101-103, generating a summoning command based on the message content, and summoning information from the corresponding message source or message-generating system.

[0142] Following step 103, messages in the processing queue are processed in groups based on the message itself, or the message and message recall information. It can also be determined whether to process messages in a message group in the processing queue based on the message status. Specifically, if all messages in the message group in the processing queue are in a completed state, the messages in that message group are processed.

[0143] In response to the fact that the status of at least one message in the message group in the pending queue is not the completed state, it is determined to put the message group back into the pending queue, continue to wait for a preset time, re-determine the status of the messages in the message group, until the status of all messages in the message group is the completed state, and then process the messages in the message group.

[0144] In particular, to prevent a message from remaining incomplete and causing messages in its corresponding message group to remain unprocessed, thus affecting the analysis of the event, the messages in the message group can be processed when the waiting time exceeds the corresponding threshold. In this way, even if there are messages in the message group that are incomplete, the messages in the message group will still be processed to ensure the timeliness of the event analysis.

[0145] The following example illustrates the message processing method described in steps 101-103. Please refer to it. Figure 3 In this example, the message is proactively reported each time it is generated.

[0146] After receiving a message, the message processing system determines whether it requires immediate response based on the message's type information, and therefore whether it needs to be grouped. If it is determined that the message requires immediate response, then the message does not need to be grouped, and thus the message is processed accordingly. The rules for this processing can be determined through business rules.

[0147] If it's determined that a message doesn't require immediate processing but needs to be grouped, then the message's type information is determined, and the message is placed in the first queue. This first queue is a delayed queue. Furthermore, based on the type information, delayed dequeue rules need to be set for the messages in this first queue. These dequeue rules correspond one-to-one with the messages in the first queue and are set to blocking take. The dequeue rule indicates the allowed delay time for the message, and within that allowed delay time, the messages that the message needs to wait for to be dequeued after it. These dequeue rules correspond to business rules, or they can be predefined based on the type of events monitored by the message source. Therefore, the messages that need to wait as indicated in the dequeue rule are those that need to be processed together with the message itself, facilitating a comprehensive analysis of the corresponding events.

[0148] Based on the aforementioned dequeue rules, messages in the first queue are dequeued and placed into the queue to be grouped.

[0149] Then, the messages in the first queue are stored in Redis. In Redis, the key is the message type, and the value is the message itself; thus, the same key can correspond to multiple messages.

[0150] Next, multiple messages are periodically retrieved from Redis, categorized by key. For messages with the same key (i.e., messages with the same type information), the message received earliest is designated as the reference message.

[0151] Determine if the current system time is greater than the sum of the reference message reception time and the longest waiting time interval. If not, determine not to group messages for now, identify messages with the same key as an intermediate message group, and add subsequent periodically retrieved messages with the same key from Redis to this intermediate message group until the current system time is greater than the sum of the aforementioned times. At this point, determine to start grouping messages in the intermediate message group with the same key.

[0152] If so, sort the messages in the intermediate message group with the same key according to their reception time to obtain the message sequence. Traverse the messages in the message sequence, determine the timestamp carried in each message, and if the time difference between the timestamp of the current message and the timestamp carried in the reference message is less than a preset time interval, determine that the current message and the reference message are a group.

[0153] After iterating through the message sequence to obtain a message group, the reference message can be re-determined for the messages in the message sequence, and the next message group can be obtained according to the aforementioned method... This process is repeated to obtain multiple message groups. After determining the message group, the messages in the message group are deleted from Redis.

[0154] Next, the message group is placed into a blocking queue for asynchronous thread processing. Message groups are periodically retrieved from this blocking queue using a non-blocking poll message retrieval method. The retrieval order of the message groups is determined based on the timeliness requirements of the events corresponding to the message groups (which can be determined according to the aforementioned business rules or predefined), and / or the time when the message groups entered the queue.

[0155] For each message group extracted from the pending queue, iterate through the status of the messages within the group. Determine whether the status of each message in the group is complete.

[0156] If not, add the messages in the message group to the pending queue.

[0157] If so, after performing processes such as format conversion, aggregation, and padding on the messages in the message group, the resulting information is used as parameters for the events corresponding to each message group. Using these parameters, analysis and calculations are then performed on the events corresponding to the message groups.

[0158] It should be noted that the queues in the above examples are all blocking queues, and the message storage and retrieval strategies of each blocking queue are different. For example, the first queue uses blocking take, while the queue to be processed uses non-blocking poll.

[0159] Based on the same inventive concept, this application provides a message processing device, which is similar to the aforementioned... Figure 1 The message processing method shown corresponds to the specific implementation of this device, which can be found in the description of the aforementioned method embodiments. Repeated descriptions will not be repeated here. Figure 4 The device includes:

[0160] Type unit 401 is used to determine the type information of the message in response to receiving the message.

[0161] The type information includes the relevance between the message and the business scenario of the message.

[0162] Type unit 401 is specifically used for:

[0163] In the message, a first object is identified based on a first target key; wherein the first object includes the first target key and a first target value, the first target value indicating the information of the business scenario and the relevance; in a preset first mapping relationship, the type information corresponding to the first target value is queried; wherein the first mapping relationship includes the correspondence between the first field and the type information.

[0164] The type unit 401 is also used for:

[0165] In the message, the first object and the second object are identified based on the first target key and the second target key; wherein, the second object is composed of the second target key and the second target value, and the second target value indicates the business rule corresponding to the business scenario; in the preset second mapping relationship, the type information corresponding to the first target value and the second target value is queried.

[0166] Grouping unit 402 is used to group messages with the same type information into multiple message groups based on the timestamps carried by the messages.

[0167] The timestamps of the messages in the message group are located within the same preset time period.

[0168] Grouping unit 402 is specifically used for:

[0169] Read the timestamp in the message; wherein the timestamp corresponds to the generation time of the message; among the messages with the same type information, determine the message corresponding to the earliest target timestamp as the reference message; determine the time when the reference message is obtained as the first time, and determine the second time based on the first time and the obtained longest waiting time threshold; determine the message group based on the first time window; wherein the start time of the first time window is the first time and the end time is the second time.

[0170] Grouping unit 402 is specifically used for:

[0171] The messages whose acquisition time falls within the first time window are identified as intermediate message groups; for the intermediate message groups, a third time is determined based on the target timestamp and the preset interval of the acquired messages; in the intermediate message groups, the messages whose timestamps fall between the target timestamp and the third time are identified as a group of messages.

[0172] The device further includes a thread unit, used to determine the target thread pool corresponding to the message; using the threads in the target thread pool, the message is added to a first queue, and a dequeue rule is determined for the message in the first queue, so that the message is dequeued from the first queue based on the dequeue rule and enters the message queue to be grouped;

[0173] The grouping unit 402 is further used for:

[0174] For messages with the same type information, the messages in the message queue to be grouped are grouped based on the timestamp carried by the message to obtain the multiple message groups.

[0175] The message is taken in a blocking manner when it is dequeued from the first queue.

[0176] The caching unit 403 is used to cache the message group to the queue to be processed.

[0177] The device further includes a status unit, configured to process messages in the message group in response to the fact that all messages in the message group in the pending queue are in the completed state; or, in response to the fact that at least one message in the message group in the pending queue is not in the completed state, wait for a preset time, determine the status of messages in the message group, and process messages in the message group until all messages in the message group are in the completed state.

[0178] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this invention, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0179] After introducing the message processing method and apparatus according to exemplary embodiments of the present invention, the electronic device according to another exemplary embodiment of the present invention will be introduced next.

[0180] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0181] Please see Figure 5 Based on the same technical concept, embodiments of this application also provide a computer device. In one embodiment, as shown in the figure, the computer device may include a memory 501, a communication module 503, and one or more processors 502.

[0182] The memory 501 is used to store computer programs executed by the processor 502. The memory 501 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, and the data storage area may store various operation instruction sets, etc.

[0183] Memory 501 may be volatile memory, such as random-access memory (RAM); memory 501 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 501 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 501 may be a combination of the above-mentioned memories.

[0184] Processor 502 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 502 is used to implement the above-described message processing method when calling computer programs stored in memory 501.

[0185] The communication module 503 is used to communicate with terminal equipment, site equipment or other network equipment.

[0186] This application embodiment does not limit the specific connection medium between the memory 501, communication module 503, and processor 502 described above. This application embodiment... Figure 5 The memory 501 and the processor 502 are connected via a bus 504, and the bus 504 is in Figure 5 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 504 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 5 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0187] The memory 501 stores a computer storage medium, which stores computer-executable instructions for implementing the message processing methods of the embodiments of this application. The processor 502 is used to execute the message processing methods of the above embodiments.

[0188] This invention also provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, the computer program being used to cause a computer to execute the technical solution of the message processing method described in the above embodiments.

[0189] This invention also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it enables the computer to implement the technical solution of the message processing method described in the above embodiments when executing the computer program.

[0190] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program instructions are computer programs, which can be stored in a computer-readable storage medium. When executed, the computer program performs the steps of the above method embodiments. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0191] The program product of embodiments of the present invention may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a computing device. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0192] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0193] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0194] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the client computing device, partially on the client device, as a standalone software package, partially on the client computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the client computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0195] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0196] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0197] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0198] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0199] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A message processing method, characterized in that, include: In response to receiving a message, the type information of the message is determined; wherein, the type information includes the relevance between the message and the business scenario of the message, and determining the type information of the message includes: in the message, based on a first target key, identifying a first object, the first object including the first target key and a first target value, the first target value indicating the information of the business scenario and the relevance; In a preset first mapping relationship, the type information corresponding to the first target value is queried; wherein, the first mapping relationship includes the correspondence between the first target value and the type information; For messages with the same type of information, the messages are grouped based on the timestamps carried by the messages to obtain multiple message groups, including: reading the timestamps in the messages; wherein the timestamps correspond to the generation time of the messages, and the timestamps of the messages in the message group are located within the same preset time period; Among the messages with the same type information, the message corresponding to the earliest target timestamp is determined to be the reference message; The moment when the reference message is obtained is determined as the first moment, and the second moment is determined based on the first moment and the obtained longest waiting time threshold; The message group is determined based on a first time window; wherein the start time of the first time window is the first time and the end time is the second time. The message group is cached in the queue to be processed.

2. The method as described in claim 1, characterized in that, The determination of the message group based on the first time window includes: The messages whose arrival time falls within the first time window are identified as the intermediate message group; For the intermediate message group, a third moment is determined based on the target timestamp and the preset interval of the acquired messages; In the intermediate message group, messages whose timestamps are located between the target timestamp and the third time point are identified as a group of messages.

3. The method as described in claim 1, characterized in that, The determination of the message type information includes: In the message, the first object and the second object are identified based on the first target key and the second target key; wherein, the second object includes the second target key and the second target value, and the second target value indicates the business rule corresponding to the business scenario; In the preset second mapping relationship, query the type information corresponding to the first target value and the second target value.

4. The method as described in claim 1, characterized in that, Before grouping messages with the same type of information based on the timestamp carried by the message, the method further includes: Determine the target thread pool corresponding to the message; Using threads in the target thread pool, the message is added to the first queue, and a dequeue rule is determined for the message in the first queue, so that the message is dequeued from the first queue according to the dequeue rule and enters the message queue to be grouped. Then, for messages with the same type information, the messages are grouped based on the timestamps they carry, resulting in multiple message groups, including: For messages with the same type information, the messages in the message queue to be grouped are grouped based on the timestamp carried by the message to obtain the multiple message groups.

5. The method as described in claim 4, characterized in that, The message is taken in a blocking manner when it is dequeued from the first queue.

6. The method as described in claim 1, characterized in that, After caching the message group to the queue to be processed, the method further includes: In response to the fact that all messages in the message group in the pending queue are in a completed state, process the messages in the message group; or, In response to a situation where at least one message in the message group in the pending queue is not in the completed state, a preset time is waited to determine the state of the messages in the message group until all messages in the message group are in the completed state, and then the messages in the message group are processed.

7. A message processing device, characterized in that, include: A type unit is used to determine the type information of a message in response to receiving a message; wherein the type information includes the relevance between the message and the business scenario of the message, and determining the type information of the message includes: in the message, identifying a first object based on a first target key, the first object including the first target key and a first target value, the first target value indicating the information of the business scenario and the relevance; In a preset first mapping relationship, the type information corresponding to the first target value is queried; wherein, the first mapping relationship includes the correspondence between the first target value and the type information; A grouping unit is used to group messages with the same type information into multiple message groups based on the timestamps carried by the messages, including: reading the timestamps in the messages; wherein the timestamps correspond to the generation time of the messages, and the timestamps of the messages in the message group are within the same preset time period; Among the messages with the same type information, the message corresponding to the earliest target timestamp is determined to be the reference message; The moment when the reference message is obtained is determined as the first moment, and the second moment is determined based on the first moment and the obtained longest waiting time threshold; The message group is determined based on a first time window; wherein the start time of the first time window is the first time and the end time is the second time. A caching unit is used to cache the message group to a queue to be processed.

8. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps of the method according to any one of claims 1-6 in accordance with the obtained program instructions.

9. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-6.

Images