Message queue consumption method, lightweight computing engine, electronic device, and storage medium

Abstract

The message queue consumption method, the lightweight computing engine, the electronic device and the computer readable storage medium are provided by the disclosure, the message queue consumption method comprises: obtaining a plurality of consumption tasks, the plurality of consumption tasks belong to a plurality of different data sources;According to the information of each data source, a plurality of data sources are aggregated into at least one virtual data source, and a consumption task set under the virtual data source is obtained;The consumption task set is regarded as a target consumption task, at least one thread is allocated to the target consumption task, and the message queue is consumed. The total thread resource overhead can be greatly reduced, and the carrying capacity of the computing engine is stronger.

Description

Technical Field

[0001] This disclosure relates to message queue technology, and more particularly to a message queue consumption method, a lightweight computing engine, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Message queues (MQ) are a communication method between applications. Applications can communicate by reading and writing messages stored in the message queue. Message producers can generate messages and send them to the message queue. The message queue can store messages until they are consumed by consumers. Message consumers can retrieve messages from the message queue and process them (subscribe to and consume messages). Topics in the message queue can represent message categories or channels. Producers can publish messages to topics, and consumers can subscribe to and consume messages from topics.

[0003] A consumer's consumption of a topic can be viewed as a consumption job. Each topic corresponds to a data source. Same-source consumption jobs are those on the same topic, while non-same-source consumption jobs are those on multiple different topics. Related technologies use a computing engine to perform the computation for consuming messages from the message queue. Each consumption job on the computing engine incurs framework overhead, such as framework threads. The more consumption jobs there are, the greater the framework overhead. Since each topic in these technologies requires at least one consumption job, and due to the limitations of the message queue mechanism, a single consumption job can only consume a single topic and cannot support cross-topic consumption. However, if a consumer needs to subscribe to and consume messages from different topics, a large number of non-same-source consumption jobs will be generated. With a fixed total thread resource and multiple non-same-source consumption jobs potentially requiring simultaneous allocation of thread resources, the thread resource overhead of a large number of non-same-source consumption jobs becomes significant, making it difficult for the computing engine to handle the load. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides a message queue consumption method, a lightweight computing engine, an electronic device, and a computer-readable storage medium. The technical solution is as follows:

[0005] According to a first aspect of the embodiments of this specification, a message queue consumption method is provided, the method comprising:

[0006] Multiple consumption tasks are obtained, and the multiple consumption tasks belong to multiple different data sources;

[0007] Based on the information from each of the data sources, multiple data sources are aggregated into at least one virtual data source to obtain a set of consumption tasks under the virtual data source;

[0008] The set of consumption tasks is treated as a target consumption task, and at least one thread is assigned to the target consumption task to consume the message queue.

[0009] According to a second aspect of the embodiments of this specification, a lightweight computing engine is provided, the lightweight computing engine being built on RocketMQ, the lightweight computing engine being used to perform the method as described in the first aspect.

[0010] According to a third aspect of the embodiments of this specification, an electronic device is provided, the electronic device comprising:

[0011] processor;

[0012] Memory used to store processor-executable instructions;

[0013] The processor is configured to implement the method as described in the first aspect.

[0014] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described in the first aspect.

[0015] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0016] In the embodiments of this specification, when consumers need to subscribe to and consume messages from different topics, multiple different data sources belonging to multiple non-same-origin consumption tasks can be aggregated into a virtual data source. Based on this virtual data source, multiple small non-same-origin consumption tasks can be merged into a large target consumption task for processing. This allows cross-topic consumption to be achieved through the same target consumption task. When there are a large number of non-same-origin consumption tasks, the total amount of resources remains unchanged, and multiple non-same-origin consumption tasks may need to allocate resources simultaneously, aggregating multiple non-same-origin consumption tasks into one task can greatly reduce the total number of tasks. At the same time, when allocating threads, they are allocated on a per-target consumption task basis, and the thread allocation is flexible based on the amount of data in the aggregated shards of the target consumption task, rather than allocating them on a per-unit basis based on a large number of individual consumption tasks before aggregation. This can greatly reduce the total thread resource overhead and make the computing engine more capable of handling the load.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments or related technologies of this specification, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 This is a schematic diagram of a message queue scenario related to the technology;

[0020] Figure 2 This is a flowchart illustrating a message queue consumption method according to one embodiment of this specification.

[0021] Figure 3 This is a schematic diagram of thread allocation according to one embodiment of this specification;

[0022] Figure 4 This is a schematic diagram of the structure of a lightweight computing engine according to one embodiment of this specification;

[0023] Figure 5 This is a schematic diagram of the structure of an electronic device according to one embodiment of this specification. Detailed Implementation

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0025] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0026] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0027] Please see Figure 1 Message queue 102 is a communication method between applications. Applications can communicate by reading and writing messages stored in the message queue. Message queue 102 allows independent applications to communicate through asynchronous message passing, thereby reducing coupling. Application messages are continuously stored in message queue 102 until they are consumed, ensuring that messages are not lost. Message producer 101 can generate messages and send them to message queue 102. Message queue 102 can store messages until they are consumed. Message consumer 103 can retrieve messages from the message queue and process them (subscribe to and consume messages).

[0028] It is worth noting that message producer 101, message queue 102, and message consumer 103 are not limited to... Figure 1 The number of message producers 101, message queues 102, and message consumers 103 is not limited.

[0029] The topics in message queue 102 are the message classification and subscription mechanism. The topics in message queue 102 can represent message categories or channels. Message producers 101 can publish messages to specific topics, and message consumers 103 can subscribe to and consume messages from topics of interest, thereby allowing applications to classify data flow according to business logic and implement the publish-subscribe model.

[0030] Message consumer 103's consumption of a single topic can be considered a consumption task. Each topic corresponds to a data source. Same-source consumption tasks are those on the same topic, while non-same-source consumption tasks are those on multiple different topics. Related technologies utilize a computation engine to perform the computation for consuming messages from the message queue. Each consumption task on the computation engine incurs framework overhead, such as framework threads. The more consumption tasks there are, the greater the framework overhead. Because each topic in these technologies requires at least one consumption task, and due to the limitations of the message queue mechanism, the same consumption task can only consume a single topic, and cross-topic consumption is not supported. In a homogeneous scenario, there may be a large number of consumption tasks targeting the same data source. Since the data source is the same, these large number of homogeneous consumption tasks can be treated as a single consumption task and will not cause the computing engine to be overwhelmed by the large number of tasks. However, if consumers need to subscribe to and consume messages from different topics, a large number of non-homogeneous consumption tasks will be generated when there are a large number of topics. With the total amount of resources remaining unchanged and multiple non-homogeneous consumption tasks may need to allocate resources at the same time, the resource overhead of a large number of non-homogeneous consumption tasks from different data sources is large, making it difficult for the computing engine to handle.

[0031] Taking the consumption scenario of security log messages as an example, when the message producer is a user of certain products (such as network devices, servers, applications, etc.), the message generated by the user can be the security log message of that product. The security log message generated by the user enters the message queue. The message consumer can be a system that can perform security log management. This system can subscribe to and consume the security log message generated by the user from the message queue in order to realize the security log management of the user's product.

[0032] Taking a message consumer-based Security Information and Event Management (SIEM) system as an example, this system is primarily used to maintain enterprise network security and ensure enterprise data security. SIEM's main functions include security log management, security incident detection, threat analysis, security response, and security auditing. Security log management mainly involves collecting, analyzing, storing, and managing security logs from network devices, servers, and applications to facilitate user auditing, diagnosis, and response to security incidents.

[0033] The SIEM system includes a data acquisition module. This data acquisition involves generating different acquisition tasks (or consumption tasks) for different data sources. It collects data from various product categories required by the user, processes, analyzes, and performs calculations before finally writing the data to a designated location within the SIEM system. The data generated by the data acquisition module forms the foundation for other modules within the SIEM system, such as data analysis, serving the subsequent detection, analysis, and response to security incidents. Therefore, data acquisition is a crucial foundation of the SIEM system; and the scheduling of data acquisition tasks is related to issues such as data integrity, task stability, and optimal resource consumption.

[0034] SIEM's data collector employs the lightweight real-time computing engine RStream (Rocket Message Queue Streams, a streaming processing system for message queues) as the task engine for data collection. This engine is compatible with SQL (Structured Query Language) syntax and supports data source types such as SLS (Log Service), Kafka (Apache Kafka, a distributed stream processing platform and message queue system), and OpenAPI (OpenAPI Specification, a specification for machine-readable interface documentation). RStream not only supports data collection but also allows for complex data processing during the collection process. Currently, RStream instances run in process mode, while collection tasks run as threads on RStream instances; RStream instances support horizontal dynamic scaling. The SIEM system's data collection tasks utilize message queue systems, such as RocketMQ (Rocket Message Queue), and also implements the RStream computing framework, a lightweight open-source real-time computing framework built on RocketMQ.

[0035] It is worth noting that the above description of message consumption scenarios is only an illustrative example. In actual applications, other application scenarios may exist, but no specific limitations are made.

[0036] To address the aforementioned issues, this specification provides a message queue consumption method to significantly reduce overall resource overhead and enhance the computing engine's capacity. Figure 2 As shown, the method includes the following steps:

[0037] S201. Obtain multiple consumption tasks, wherein the multiple consumption tasks belong to multiple different data sources;

[0038] It should be noted that the above multiple consumption tasks correspond to different topics in the message queue. In other words, these multiple consumption tasks are non-same-source consumption tasks, and each of them belongs to a different data source.

[0039] As an example, each consumption task corresponds to a data source, and each consumption task can correspond to a message queue. Multiple consumption tasks can correspond to different topics of the message queue.

[0040] As an example, a topic, as an implementation of a message queue, can take different forms depending on the type of message queue. For instance, a topic can be directly called a topic in one type of message queue, a topic in another, and a logstore in yet another. It is worth noting that the above description of the implementation of topics in different message queues is merely an illustrative example. In practical applications, other implementations may exist, but no specific limitations are imposed on them.

[0041] To illustrate the topic, suppose the message consumer subscribes to the alert log messages of the external gateway (web application firewall, WAF), then the WAF alert logs can be a topic. Suppose the message consumer subscribes to the raw log messages of the WAF, then the raw logs of the WAF can be another topic.

[0042] As an example, a message queue can carry one topic or multiple topics, without any specific limitation.

[0043] As an example, a topic's attributes can include a topic name, which can be a unique identifier for the topic. This identifier can contain letters, numbers, or other symbols, and there are no specific restrictions on the topic name.

[0044] As an example, in data acquisition scenarios, data sources can be of various types, such as the aforementioned online real-time data SLS or Kafka, and offline data such as databases (DB) or OpenAPI. As another example, in a cloud computing scenario, a user may configure multiple cloud products of various types, each of which can serve as a data source. Taking WAF (Web Application Firewall Cloud Product) as an example, if a user starts a WAF cloud product instance and needs to obtain security services from that instance, the WAF cloud product instance can access the SIEM system. From a business perspective, this will generate a data acquisition task, or consumption task, of the logs from that WAF cloud product instance.

[0045] In some scenarios, data from the data source is stored in a database, where sharding can be configured by the user or dynamically generated as needed. For example, WAF logs are stored in a log store, which can serve as a data source. A cloud product may have multiple log stores, which are considered multiple sources.

[0046] S202. Based on the information of each data source, aggregate multiple data sources into at least one virtual data source to obtain a set of consumption tasks under the virtual data source;

[0047] For example, based on the information from each data source, multiple data sources can be aggregated into at least one virtual data source to obtain a set of consumption tasks corresponding to each virtual data source.

[0048] In related technologies, since one data source corresponds to one topic of a consumption task, i.e., the data source is fixed to one topic, the fixed calculation logic in the consumption task calculation framework of related technologies is to consume data as an independent unit when consuming data. Therefore, in order to break through this calculation logic, multiple small data sources can be aggregated into at least one large virtual data source, so that data can be consumed as an independent unit when consuming data. Based on the virtual data source, the consumption tasks corresponding to multiple different data sources contained therein can be merged into a consumption task set.

[0049] Considering that there may be multiple data sources under the same user producing messages, and there may also be multiple users producing messages, if multiple data sources from different users are aggregated into a single virtual data source, and threads are allocated to the set of consumption tasks under this virtual data source, then if the allocated thread crashes for some reason, it may affect the consumption tasks from different users under this virtual data source, resulting in a high risk of system isolation.

[0050] To address the aforementioned issues, system isolation risks can be mitigated in various ways. For example, one approach could include: obtaining the user identifier of the data source corresponding to each consumption task. This user identifier can be used to represent the target user of the produced message. Multiple data sources with the same user identifier are aggregated into at least one virtual data source. That is, the multiple consumption tasks can correspond to the same target user. Aggregating the data sources corresponding to multiple consumption tasks of the same target user into at least one virtual data source, where the target user is the current user, will only affect the multiple consumption tasks of the same target user (i.e., the current user) from which the virtual data source originates, even if the thread allocated to the set of consumption tasks under this virtual data source crashes. It will not affect the consumption tasks of other users besides the current user, thus balancing lower system isolation risks with better processing performance. It is worth noting that the above description of multiple data sources corresponding to the same user identifier is only an illustrative example. In actual applications, other correspondences may exist. For example, the data sources corresponding to the multiple consumption tasks may also have different user identifiers, but the number of users corresponding to these different user identifiers is less than or equal to a preset number. This preset number can be 2, 3, or other numbers. The specific preset number is not limited. Limiting the number of different users corresponding to multiple data sources to the preset number can minimize the scope of affected users in the event of a system crash in which the computing engine is located. Therefore, the specific correspondence is not limited.

[0051] To illustrate the above embodiment, consider the following example: Assume there are two users, User 1 and User 2, each with multiple data sources. These users correspond to multiple non-same-source consumption tasks. While it's possible to aggregate the data sources corresponding to User 1 and User 2 into a single virtual data source and allocate threads to the consumption task set under this virtual data source, if the thread allocated to this virtual data source crashes, both User 1's and User 2's consumption tasks may be affected. To ensure lower isolation risk, if User 1 is the current user, aggregation can be performed only within the scope of the multiple data sources corresponding to User 1, without adding User 2's data source to the aggregation of User 1's data source. Then, if the thread allocated to the consumption task set under User 1's virtual data source crashes, while the thread allocated to the consumption task set under User 2's virtual data source may not crash, only the current user (User 1)'s consumption tasks will be affected, not User 2's. In other words, each thread's consumption task originates from one user; when one thread crashes, only the consumption tasks of that user are affected.

[0052] As an example, the user identifier mentioned above can be the target user's account, and there are no restrictions on the specific form of the user identifier.

[0053] Considering that the data sources corresponding to the above multiple consumption tasks have the same user identifier, that is, multiple data sources correspond to the same target user, there may be multiple dimensions under the target user, and different dimensions may correspond to different data sources. If we simply aggregate all the data sources corresponding to the target user into at least one virtual data source and allocate threads to the set of consumption tasks under the virtual data source, then the stability risk of the user dimension is still high if the allocated thread crashes.

[0054] To address the aforementioned issues, several methods can be used to further reduce the stability risks associated with the user dimension. For example, one approach could include: obtaining the user dimension identifiers of multiple data sources sharing the same user identifier. These identifiers characterize the target dimension to which the messages produced by the target user belong. Multiple data sources with the same user dimension identifier are then aggregated into at least one virtual data source. Furthermore, assuming that the data sources corresponding to the multiple consumption tasks share the same user identifier (i.e., the multiple consumption tasks correspond to the same target user), these data sources can also share the same user dimension identifier. Aggregating the data sources corresponding to the multiple consumption tasks with the same target dimension into at least one virtual data source, with this target dimension serving as the current dimension, and allocating threads to the set of consumption tasks under this virtual data source, will only affect the consumption tasks under the target dimension (current dimension) of this virtual data source, without impacting consumption tasks in other dimensions. This further reduces the stability risks associated with the user dimension.

[0055] To illustrate the above embodiments: Assume the same user identifier is the account of the target user. This target user can have three dimensions: sub-accounts, distribution regions, and products. Assuming there are 200 sub-accounts, 5 regions, and 22 products, the total number of data sources corresponding to this target user could be: sub-accounts * regions * products = 22,000. While it's possible to aggregate all 22,000 data sources into a single virtual data source, this virtual data source has high stability risks across all dimensions and low overall processing performance. Therefore, data sources within the same dimension can be aggregated. For example, data sources with the same products can be aggregated to obtain 22 virtual data sources, or data sources with the same regions and products can be aggregated to obtain 110 virtual data sources. This reduces the stability risks across all dimensions while maintaining good processing performance for the aggregated virtual data sources. It is worth noting that the above description of aggregation methods for the same dimension is merely an illustrative example. In practical applications, other aggregation methods for the same dimension may exist, as well as dimensions other than those described above. No specific limitations are imposed on this.

[0056] S203. Take the set of consumption tasks as a target consumption task, and allocate at least one thread to the target consumption task to consume the message queue.

[0057] For example, each set of consumption tasks can be treated as a target consumption task, and at least one thread can be assigned to that target consumption task to consume the message queue.

[0058] As an example, the aforementioned threads could be framework overhead on the computing engine, such as framework threads.

[0059] This embodiment can allocate threads in various ways. As an example, one allocation method may include:

[0060] Based on the target consumption task, determine the data shards of the data source to which each consumption task in the corresponding consumption task set belongs, thus obtaining the total number of data shards. For example, determine the data shards of the data source corresponding to each consumption task in the corresponding consumption task set, thus obtaining the total number of data shards. Allocate at least one thread to the target consumption task based on this total number of data shards. It should be noted that data shards are contained within topics, and the total number of data shards can refer to the total number of data shards aggregated from different topics. Thread allocation can be flexibly performed based on the size of the total number of data shards.

[0061] This embodiment can allocate threads based on the total number of data shards in various ways. For example, one allocation method may include: first, setting a total threshold; if the total number of data shards is greater than or equal to the total threshold, then allocating a first number of threads to the target consumption task; if the total number of data shards is less than the total threshold, then allocating a second number of threads to the target consumption task. The first number is greater than the second number, that is, when the total number of data shards is large, more threads are allocated to the target consumption task, and when the total number of data shards is small, fewer threads are allocated to the target consumption task, so that the allocated threads can flexibly adapt to the total number of data shards, thereby reducing the waste of thread resources. It is worth noting that the above description of setting a total threshold for allocation is only an illustrative example. In practical applications, other allocation methods may exist. For example, several reference intervals for the total number of data shards can be set first. These reference intervals can each correspond to a specific number of threads. The total number of data shards can be compared with these reference intervals. The specific number of threads corresponding to the reference intervals for the total number of data shards that the total number of data shards falls into can be allocated to the target consumption task. Therefore, there are no specific limitations on how to allocate threads based on the total number of data shards.

[0062] It's worth noting that the above description of thread allocation based on the total number of data shards is merely an illustrative example. In practical applications, other thread allocation methods may exist. For instance, the total number of data shards may not be considered; instead, a preset number of threads may be allocated to the target consumption task based on demand. For example, when the lowest possible overall thread usage is required, the preset number of threads can be small; when faster overall processing performance is needed, the preset number of threads can be large. Even if the total number of data shards is small, the preset number of threads can be large; even if the total number of data shards is large, the preset number of threads can be small. Therefore, there are no specific limitations on how thread allocation should be performed.

[0063] In related technologies, message queue consumption is typically based on a push model. Due to the push model mechanism, there's an upper limit to the number of consumer groups a single user can have (e.g., a user can only have a maximum of 32 consumer groups). This limits the number of concurrent threads that can be allocated to the same target consumption task, making it impossible to exceed this limit. When using the push model for message transmission in a message queue, the message producer actively pushes messages to the message queue, and the message consumer receives messages by subscribing to the message queue of the relevant topic. The push model involves subscribing to a data source to collect data. However, the push model also has a limit on the number of consumer groups a data source can have. A consumer group refers to a group of message consumers that collectively consume messages from a queue on the same topic. The limit on the number of consumer groups refers to the number of data sources that can be used; for example, some data sources may limit the number of consumers to 2000. In practical applications, if there are many different business processes that need to consume the same data source, once the limit on the number of consumer groups for that data source is reached, no more consumer groups can be created, preventing the business from using the data source.

[0064] To address the aforementioned issues, the limitation on the number of consumer groups in the Push mode can be overcome in several ways. For example, one approach could be to allocate threads to a target consumption task, and have each thread pull messages from the message queues of the various consumption tasks under the target consumption task for processing, thereby consuming the message queues.

[0065] As an example, the process of pulling messages for processing described above can be implemented using Pull mode. This means changing the message queue mode from Push mode to Pull mode. Due to the inherent mechanism of Pull mode, message consumers actively pull messages from the message queue of the corresponding topic for processing. There is no limit to the number of consumer groups, thus bypassing the limit on the number of consumer groups. This allows for an unlimited number of concurrent threads allocated to the target consumption task, which can be flexibly configured as needed. It is worth noting that the above description of using Pull mode to overcome the limit on the number of consumer groups is merely an example. In practical applications, other methods to overcome the limit on the number of consumer groups may exist, and these are not specifically limited here.

[0066] In related technologies, consumption tasks are expressed in the form of Structured Query Language (SQL), meaning one SQL statement represents one consumption task. The framework of the computing engine in these technologies is based on a consumption task, and each consumption task represents a business logic. If multiple small consumption tasks are forcibly merged into one large consumption task in the computing engine of these technologies, that is, multiple small SQL statements are merged into one large SQL statement, then multiple small SQL statements need to be placed in a large SQL file, which will greatly increase the operation and maintenance costs. For example, if a small SQL statement has 100 lines, then 10 small SQL statements are merged into one large SQL statement, which has 1000 lines. The operation and maintenance cost of a 1000-line SQL statement is undoubtedly huge.

[0067] To address the aforementioned issues, the operational costs of the merged SQL statements can be reduced in various ways. For example, one approach could be to implement the message queue consumption method described in any of the embodiments above based on a lightweight computing engine built with RocketMQ. The lightweight computing engine built with RocketMQ allows for independent operation and maintenance management and independent testing of each SQL statement within a single SQL statement merged from multiple SQL statements, significantly reducing the overall operational costs.

[0068] The message queue consumption method described in any of the embodiments above can be applied to various scenarios. For example, the message queue consumption method described in any of the embodiments above can be applied to the consumption of security log messages, enabling the security log management system to consume security log messages in the message queue more efficiently. This better facilitates the collection, analysis, and storage of security logs from various devices (such as network devices, servers, etc.), thereby enabling users to audit, diagnose, and respond to security events. It is worth noting that the above descriptions of application scenarios are merely illustrative. In practical applications, other application scenarios are possible, such as private cloud, industrial intelligence, and the Internet of Things (IoT). Therefore, the specific application scenarios of the embodiments in this specification are not limited.

[0069] As an example, when the message queue consumption method described in any of the embodiments above is applied to a security log message consumption scenario, the consumption resources allocated to the target consumption task can also be the memory occupied by the dimension table, which can be obtained based on a query of the security log message.

[0070] The message queue in this embodiment may include RabbitMQ (Rabbit Message Queue, a message middleware), Kafka, ActiveMQ (Active Message Queue, a message middleware), RocketMQ, etc. Taking RocketMQ as an example, it includes four parts: NameServer (naming service) cluster, Broker (broker) cluster, producer cluster, and consumer cluster; wherein:

[0071] The NameServer cluster implements a service discovery mechanism to manage information about all Brokers and Topics within the cluster. Key functions of the NameServer include Topic routing management, Broker management (including Broker registration and heartbeat detection), and whitelist management of message senders and consumers.

[0072] A broker cluster is used for storing and forwarding messages. A broker cluster can consist of a single server, with each server running one or more broker instances. The broker instances are primarily responsible for message storage, delivery, and consumer retrieval operations. In a broker cluster, all message queues are distributed across different brokers, achieving distributed data storage and load balancing. The broker cluster can also implement a master-slave mode. In this mode, each topic's master broker has one or more slave brokers. Slave brokers retrieve all data from the master broker and synchronize in real time. When the master broker fails or crashes, a slave broker can be quickly promoted to master, ensuring message persistence and preventing data loss.

[0073] A producer cluster typically consists of multiple producer instances, each of which can send messages to a message queue. In RocketMQ, when a producer sends a message, it can choose to send it to a specified Topic and Message Queue, or it can let RocketMQ automatically select the Message Queue.

[0074] A consumer cluster typically consists of multiple consumer instances, each of which can pull messages from a specified MessageQueue for consumption. In RocketMQ, the consumer cluster coordinates message distribution and load balancing through consumer groups. Consumers can specify different consumption requirements by specifying consumption order, pull mode, and message filtering rules.

[0075] Please see Figure 3 The following describes a specific application scenario of an embodiment of this specification:

[0076] First of all Figure 3 This document introduces relevant technical solutions, taking a WAF (Web Application Firewall) data source as an example. Using user accounts as an example, one account can be a data source. In these technologies, assuming 200 accounts, there are 200 data sources, corresponding to 200 consumption tasks (jobs). Assuming each job is allocated 2 concurrent threads and 2 dimension table memory, the total number of threads and dimension table memory allocated to these 200 data sources is 400. However, in the embodiments described in this specification, these 200 data sources are aggregated into a virtual data source, and the set of consumption tasks corresponding to this virtual data source is used as a target consumption task for thread allocation. For example… Figure 3 The job concurrency in this system can be scaled on demand. That is, the consumption resources allocated to the target consumption task can be set according to requirements. If concurrency is not considered, one thread and one dimension table memory can be allocated to the target consumption task, resulting in a total of one thread and one dimension table memory. If concurrency is considered, and the amount of aggregated sharded data is small, at least two concurrent threads and two dimension table memories can be allocated to the target consumption task, resulting in a total of two threads and two dimension table memories. Alternatively, the default setting can be 100 concurrent threads and 100 dimension table memories allocated to the target consumption task, resulting in a total of 100 threads and 100 dimension table memories. Compared to solutions in related technologies, resource overhead is reduced by 75%.

[0077] In the embodiments of this specification, when consumers need to subscribe to and consume messages from different topics, multiple different data sources belonging to multiple non-same-origin consumption tasks can be aggregated into a virtual data source. Based on this virtual data source, multiple small non-same-origin consumption tasks can be merged into a large target consumption task for processing. This allows cross-topic consumption to be achieved through the same target consumption task. When there are a large number of non-same-origin consumption tasks, the total amount of resources remains unchanged, and multiple non-same-origin consumption tasks may need to allocate resources simultaneously, aggregating multiple non-same-origin consumption tasks into one task can greatly reduce the total number of tasks. At the same time, when allocating threads, they are allocated on a per-target consumption task basis, and the thread allocation is flexible based on the amount of data in the aggregated shards of the target consumption task, rather than allocating them on a per-unit basis based on a large number of individual consumption tasks before aggregation. This can greatly reduce the total thread resource overhead and make the computing engine more capable of handling the load.

[0078] Corresponding to the above method embodiments, as a specific application, the above embodiments can be implemented within a lightweight computing engine. Therefore, this specification also provides a lightweight computing engine. See [link to documentation]. Figure 4 As shown, the lightweight computing engine 401 can be built based on RocketMQ 402; the lightweight computing engine 401 is used to: obtain multiple consumption tasks, the multiple consumption tasks belonging to multiple different data sources; according to the information of each of the data sources, aggregate the multiple data sources into at least one virtual data source, and obtain a set of consumption tasks under the virtual data source; take the set of consumption tasks as a target consumption task, and allocate at least one thread to the target consumption task to consume the message queue.

[0079] As an example, the lightweight computing engine 401 is specifically used to determine the data shards of the data source to which each consumption task in the consumption task set belongs, based on the target consumption task, and to obtain the total number of data shards; and to allocate at least one thread to the target consumption task based on the total number of data shards.

[0080] As an example, the lightweight computing engine 401 is specifically used to allocate a first number of threads to the target consumption task if the total number of data shards is greater than or equal to the total number threshold; and to allocate a second number of threads to the target consumption task if the total number of data shards is less than the total number threshold, wherein the first number is greater than the second number.

[0081] As an example, the lightweight computing engine 401 is specifically used to obtain the user identifier of each of the data sources, and aggregate multiple data sources with the same user identifier into at least one virtual data source, wherein the user identifier is used to characterize the target user of the produced message.

[0082] As an example, the lightweight computing engine 401 is specifically used to obtain the user dimension identifier of each data source in multiple data sources with the same user identifier, and aggregate the multiple data sources with the same user dimension identifier into at least one virtual data source, wherein the user dimension identifier is used to characterize the target dimension to which the message produced by the target user belongs.

[0083] As an example, the lightweight computing engine 401 is specifically used to allocate threads to a target consumption task, pull messages from the message queues of the various consumption tasks under the target consumption task for processing, so as to consume the message queues.

[0084] As an example, the Lightweight Computing Engine 401 can be applied to scenarios involving the consumption of security log messages.

[0085] It is worth noting that all the embodiments in the message queue consumption method described above can be applied to the process by which the lightweight computing engine 401 aggregates multiple data sources into a single virtual data source to consume message queues.

[0086] This specification also provides an electronic device, such as... Figure 5 As shown, the electronic device includes:

[0087] Processor 501;

[0088] Memory 502 is used to store processor-executable instructions;

[0089] The processor 501 is configured to implement the message queue consumption method described in any of the embodiments above.

[0090] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the message queue consumption method described in any of the embodiments above.

[0091] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0092] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0093] The above embodiments can be applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The hardware of the computer device includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0094] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0095] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0096] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0097] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0098] The terms "specific example" or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with the embodiments or examples, which are included in at least one embodiment or example of this specification. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0099] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0100] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0101] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A message queue consumption method, comprising: Multiple consumption tasks are obtained, and the multiple consumption tasks belong to multiple different data sources; Obtain the user identifier of each of the data sources, aggregate multiple data sources with the same user identifier into at least one virtual data source, and obtain a set of consumption tasks under the virtual data source. The user identifier is used to represent the target user of the produced message. The set of consumption tasks is treated as a target consumption task, and at least one thread is assigned to the target consumption task to consume the message queue.

2. The method according to claim 1, wherein at least one thread is allocated to the target consumption task, comprising: Based on the target consumption task, determine the data shards of the data source to which the consumption task belongs in the consumption task set, and obtain the total number of data shards; At least one thread is allocated to the target consumption task based on the total number of data shards.

3. The method according to claim 2, wherein allocating at least one thread to the target consumption task based on the total number of data shards comprises: If the total number of data shards is greater than or equal to the total threshold, then a first number of threads are allocated to the target consumption task; If the total number of data shards is less than the total threshold, then a second number of threads are allocated to the target consumption task, wherein the first number is greater than the second number.

4. The method according to claim 1, wherein aggregating multiple data sources having the same user identifier into at least one virtual data source comprises: Obtain the user dimension identifier of each data source from multiple data sources that have the same user identifier, and aggregate the multiple data sources with the same user dimension identifier into at least one virtual data source. The user dimension identifier is used to characterize the target dimension to which the message produced by the target user belongs.

5. The method according to claim 1, wherein allocating at least one thread to the target consumption task to consume the message queue comprises: Each thread assigned to a target consumption task pulls messages from the message queues of each consumption task under the target consumption task for processing, in order to consume the message queues.

6. The method according to claim 1, wherein the method is implemented based on a lightweight computing engine, and the lightweight computing engine is built based on RocketMQ.

7. The method according to claim 1, wherein the method is applied to a scenario of consuming security log messages.

8. A lightweight computing engine, said lightweight computing engine being built on RocketMQ, said lightweight computing engine being used to execute the message queue consumption method according to any one of claims 1 to 7.

9. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images