A novel distributed messaging system, intelligent scheduler, device, and storage medium.

Abstract

The present invention discloses a novel distributed messaging system, comprising an intelligent scheduler and a consumer agent, each of which is located in each distributed computing node. The intelligent scheduler internally stores an intelligent scheduling algorithm for generating intelligent strategies to ensure a high message hit rate in the consumer agent while minimizing the amount of messages stored. The consumer agent internally includes an intermediary agent module for acting as a proxy, a coordinator module for instructing a data synchronization module to obtain data, a data synchronization module for obtaining data according to the coordinator's request and storing it in a small partition mirror, and a small partition mirror module internally storing data. The present invention enables applications to receive messages from the consumer agent within their node, reducing message transmission from the agent to the node where the application is running. Furthermore, since most message consumption occurs within a single node, message transmission on the network is significantly reduced.

Description

Technical Field

[0001] This invention relates to the field of message processing technology, specifically to a novel distributed messaging system, intelligent scheduler, device, and storage medium. Background Technology

[0002] The Internet of Things (IoT) technology, as a way to intelligently integrate the real and virtual worlds, has developed rapidly in recent years. By 2050, it is estimated that there will be 24 billion interconnected devices in the network, generating a large number of messages and events for communication. Messaging systems play a crucial role in data transmission in the IoT era.

[0003] The most popular messaging systems currently are publish-subscribe messaging systems, such as Kafka or Pulsar. The approach in which producers send data to brokers and consumers receive data from brokers is that each consumer establishes a connection with the broker to consume data.

[0004] Taking Kafka as an example, in existing publish-subscribe messaging system implementations, each consumer in an application establishes its own independent connection with the broker to consume data. If multiple applications want to consume the same topic, each consumer will independently retrieve data from the partition. Even if consumers are located on the same node, the same data will be sent to each consumer. This means that the same data will be sent from the same broker to the same node multiple times. Considering that many applications connect to the broker and consume data on the same topic, this sending mechanism may lead to message storms, reduce network performance, and in the worst case, may cause network outages. It will also lead to a large amount of additional network bandwidth requirements and cost consumption.

[0005] Therefore, this application proposes a novel distributed messaging system to solve the aforementioned technical problems. Summary of the Invention

[0006] The main objective of this invention is to provide a novel distributed messaging system that introduces consumer agents into nodes and uses appropriate algorithms and mechanisms to ensure that messages in the agents are transmitted to the nodes only once as much as possible. This allows applications to receive messages from the consumer agents within their respective nodes, thereby solving the technical problems mentioned in the background art and reducing message transmission from the agent to the node where the application runs.

[0007] The present invention solves the above-mentioned technical problems by adopting the following technical solutions:

[0008] A novel distributed messaging system comprises an intelligent scheduler and a consumer agent, both of which are located in each distributed computing node. The consumer agent internally includes a data synchronization module, a small partition mirroring module, a mediator module, and a coordinator module.

[0009] The intelligent scheduler internally stores and executes intelligent scheduling algorithms to generate intelligent policies and guide the coordinator module to ensure a high message hit rate in the consumer agent while minimizing the amount of messages stored.

[0010] The intermediary / agent module acts as an agent, providing services to consumers.

[0011] The coordinator module handles registrations from the intermediary agent module and has an internal smart policy to instruct the data synchronization module to obtain data from the intermediary agent module.

[0012] The data synchronization module is used to obtain the data of the topic requested by the coordinator from the intermediary agent module and store the data in a small partition image. If there are no active consumers, the data synchronization module stops or continues to obtain data according to the configuration.

[0013] A small partition mirroring module that internally stores data from the intermediary agent module, including file systems, databases, or in-memory storage.

[0014] Preferably, when a consumer subscribes to a topic or submits an offset for a topic, the intermediary agent module sends the topic, consumer group, and requested or submitted offset data to the coordinator.

[0015] Preferably, when a consumer consumes data, the intermediary agent module is used to retrieve data from a small partition mirror and respond to the consumer.

[0016] Preferably, the intelligent strategy within the coordinator module includes:

[0017] (1) Minimum: The minimum retained offset is equal to the consumer's minimum committed offset;

[0018] (2) Maximum: Data will only be deleted when the size of a small partition image reaches its configured maximum limit;

[0019] (3) Fixed offset range: When the offset range stored in the small partition image is greater than the configured maximum offset range, delete the data;

[0020] (4) Generate retention strategies based on the AI ​​intelligent strategies of the intelligent scheduler.

[0021] An intelligent scheduler, applied to any of the novel distributed messaging systems described above, executes the following specific workflow through an intelligent scheduling algorithm:

[0022] S1. Data Collection and Feature Extraction: Collect and extract access pattern features for each consumer i from the coordinator, including: access frequency f i (t), access latency requirement δ i (t), the offset range of consumption [o i，min (t), o i，max Consumer attributes A, including [t], geographical location, and device type. i Historical Access Mode H i (t);

[0023] S2. Advanced Consumer Behavior Modeling: A Long Short-Term Memory (LSTM) network model is used to model the access frequency and consumption interval of each consumer i, capturing temporal dependencies and complex patterns to predict future access frequencies. and consumption range have:

[0024]

[0025] Among them, LSTM f and LSTM o These are LSTM models for predicting access frequency and consumption intervals, respectively.

[0026] S3. Collaborative Filtering and Preference Learning: Utilizing collaborative filtering techniques, based on the similarity between consumers, to enhance the prediction of the probability of a specified offset being accessed;

[0027] S4. Multi-objective probability estimation: Combining access probability and storage cost to perform multi-objective optimization;

[0028] S5. Establish a reinforcement learning-driven optimization prefetching strategy: Use deep Q-network algorithm for deep reinforcement learning to dynamically learn and optimize the prefetching strategy;

[0029] S6. Introduction of Dynamic Adjustment and Adaptive Feedback Mechanism: An adaptive feedback mechanism is introduced to continuously monitor system performance and dynamically adjust model parameters.

[0030] Preferably, the specific operational procedure for enhancing the probability prediction of a specified offset being accessed in step S3 includes:

[0031] S31. The similarity between consumers i and j is presumably measured by the similarity function sim(i, j);

[0032] S32. Calculate consumer similarity using cosine similarity:

[0033]

[0034] Among them, f i and f j It is the access frequency vector of consumers i and j;

[0035] S33. For consumers i and j, adjust their access probabilities. and To make preference P i For consumer o, enhancements include:

[0036]

[0037] Where the consumption range of i is j's consumption range is

[0038] Preferably, the specific operational procedure for multi-objective optimization in step S4 includes:

[0039] S41. Set objectives: maximize hit rate H and minimize storage cost S;

[0040] S42. Combining the access probabilities of all consumers, calculate the total access probability. The total probability P(o, t) that a single offset consumer o is accessed at time t is:

[0041]

[0042] Where P i (o, t) represents the access probability of a single consumer;

[0043] S43. Assuming that the storage cost of each offset is the same, define the storage cost of each offset consuming o as C(o) = 1;

[0044] S44. Construct a weighted objective function as the objective function for multi-objective optimization:

[0045]

[0046] Here, α and β are weighting parameters, used to consider the balanced hit rate and storage cost, respectively.

[0047] Preferably, the method of using the deep Q-network algorithm in step S5 includes:

[0048] S51. Preset State S t Action A t and reward R t To strengthen the learning framework, in which:

[0049] S t={S(t-1), P(o,t), A i H i (t)}

[0050] State S t Includes: the offset set S(t-1) stored at the current time t, the predicted access probability P(o,t), and the consumer attribute A. i and historical access mode H i (t);

[0051] A t ={S(t)}

[0052] Consumer attribute A t Choose a new offset set S(t) for prefetching;

[0053]

[0054] Represents reward R t The reward function considers the hit rate and storage cost C(o), where λ and μ are weight parameters;

[0055] S52. Formula for updating the Q function using a deep Q-network:

[0056]

[0057] in:

[0058] Q(S t , A t ;θ) represents the current state S t Take action A t The Q value, with parameter θ;

[0059] α represents the learning rate, which is used to control the update step size;

[0060] R t This represents the immediate reward received at time point t;

[0061] γ represents the discount factor, used to weigh the importance of current rewards against future rewards;

[0062] This is represented as the next state S. t+1 The largest Q value among all possible actions;

[0063] S53. Employ ε-greedy strategies for action selection to balance exploration and exploitation:

[0064]

[0065] Where ∈ is a very small positive number.

[0066] Preferably, the specific calculation steps for the adaptive feedback mechanism to dynamically adjust the model parameters in step S6 include:

[0067] S61. Obtain the actual hit rate H of monitoring through real-time monitoring. actual (t) and storage usage S actual (t);

[0068] S62. Calculate the prediction error Error(t) and use optimization methods, including gradient descent, to adjust the prediction model parameters to reduce the error:

[0069]

[0070] Where P(o,t) represents the probability of a consumer's visit, and S(t) represents the offset set;

[0071] S63. Dynamically adjust the storage capacity C(t) based on system load and storage usage:

[0072]

[0073] Where C0 is the initial capacity, ΔC is the adjustment step size, and S... max That is the maximum storage capacity.

[0074] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the method executed by the intelligent scheduler in the novel distributed messaging system described above.

[0075] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method executed by the intelligent scheduler in the novel distributed messaging system described above.

[0076] As can be seen from the above technical solution, the present invention provides a novel distributed messaging system. Compared with the prior art, the present invention has the following advantages:

[0077] 1. This invention constructs a distributed messaging system by setting up intelligent schedulers and consumer agents at distributed computing nodes. It can seamlessly integrate with existing distributed messaging systems while simultaneously upgrading the overall messaging service functionality, thereby reducing message transmission from the agent to the node where the application runs.

[0078] 2. This invention introduces a consumer agent into the node and generates intelligent policies based on an intelligent scheduling algorithm to guide the coordinator module. This ensures that messages in the agent are transmitted to the node only once as much as possible, allowing the application to receive messages from the consumer agent within its node. At the same time, by using this method, most message consumption occurs within a single node, and message transmission over the network will be significantly reduced.

[0079] 3. This invention can significantly reduce network bandwidth consumption without changing consumer behavior and logic, while also achieving lower latency and other performance characteristics, and minimizing storage costs while maintaining a high message access hit rate, thus achieving multi-objective optimization of performance and cost.

[0080] 4. The intelligent scheduler of this invention, by combining advanced deep learning models, collaborative filtering, multi-objective optimization and reinforcement learning, can dynamically and intelligently estimate and select the message offset interval that needs to be prefetched. This not only improves the message access hit rate, but also effectively controls storage costs. It has high adaptability and scalability, achieving a virtuous balance between minimizing storage costs and maximizing access hit rate.

[0081] 5. The intelligent scheduler of this invention, through a continuous dynamic adjustment and feedback mechanism, can adaptively respond to complex and changing consumer behavior patterns, ensuring that the system can maintain efficient operation in various scenarios.

[0082] It should be understood that the descriptions in this section are not intended to identify key or essential features of embodiments of the invention, nor are they intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Of course, implementing any product of the invention does not necessarily require achieving all of the advantages described above simultaneously. Attached Figure Description

[0083] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0084] Figure 1 This is a schematic diagram of the overall modular architecture of the system of the present invention;

[0085] Figure 2 This is a schematic diagram of the structural response of the consumer agent and intelligent scheduler of the present invention;

[0086] Figure 3 This is a schematic diagram of the structural framework of the consumer agent of the present invention;

[0087] Figure 4 This is a schematic diagram of the framework structure of the Kafka system in the existing technology. Detailed Implementation

[0088] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0089] For details in the embodiments, please refer to Figures 1 to 4 .

[0090] Among existing technologies, the most popular messaging systems are publish-subscribe messaging systems, such as Kafka or Pulsar, among which... Figure 4 As shown, Kafka is used as an example for illustration, but the present invention is not limited to Kafka. In existing publish-subscribe messaging system implementations, each consumer in the application establishes its own independent connection with the broker to consume data.

[0091] like Figure 4 As shown, if multiple applications want to consume the same topic, each consumer will independently retrieve data from the partition. Even if consumers are located on the same node, the same data will be sent to each consumer. This means that the same data will be sent from the same broker to the same node multiple times. Considering that many applications connect to the broker and consume data on the same topic, this sending mechanism can lead to message storms, degrade network performance, and in the worst case, cause network outages. It also results in significant additional network bandwidth requirements and costs.

[0092] Therefore, as Figure 1 , Figure 2 and Figure 3 As shown in the embodiments of the present invention, a novel distributed messaging system is proposed. The system consists of an intelligent scheduler and a consumer agent, which are set in each distributed computing node. It can seamlessly integrate with the existing distributed messaging system and simultaneously upgrade the overall messaging service function to reduce message transmission from the agent to the node where the application runs.

[0093] By introducing consumer agents into nodes and using appropriate algorithms and mechanisms to ensure that messages in the agent are transmitted to the node only once as much as possible, applications can receive messages from the consumer agent within their own node. Furthermore, with this approach, most message consumption occurs within a single node, significantly reducing message transmission over the network.

[0094] Further, such as Figure 2 As shown, a consumer agent and a smart scheduler are created in each distributed computing node. For each application's consumer, they connect to the consumer agent and consume data from the consumer agent, rather than consuming data directly from the broker. Under the guidance of the smart scheduler, the consumer agent synchronizes data from the broker on behalf of the consumer and distributes messages to each application's consumer. The smart scheduler needs to ensure the highest message access hit rate while storing as few messages as possible.

[0095] From the application's perspective, it consumes data from the local host, not from a proxy. The consumer proxy should provide the same behavior and interface as the proxy, making it transparent to the application. Furthermore, since data is retrieved from the local host, latency and other metrics will be lower.

[0096] From the broker's perspective, the consumer broker acts as a regular consumer consuming data from it. The interface between the broker and the consumer broker remains the same as the traditional interface between the broker and the consumer, and the interface between the consumer broker and the consumer also remains the same as the traditional interface between the broker and the consumer.

[0097] The system is as follows Figure 3 As shown, the consumer agent internally includes a data synchronization module, a small partition mirroring module, an intermediary agent module, and a coordinator module, specifically including:

[0098] (a) The intermediary agent module acts as an agent to provide services to consumers. When a consumer subscribes to a topic or submits a topic offset, the intermediary agent module sends the topic, consumer group, and requested or submitted offset data to the coordinator. When a consumer consumes data, the intermediary agent module retrieves data from the small partition mirror and responds to the consumer.

[0099] (b) A coordinator module, used to handle registrations from the intermediary agent module, and internally configured with intelligent policies to instruct the data synchronization module to obtain data from the intermediary agent module. The intelligent policies within the coordinator module include:

[0100] (1) Minimum: The minimum retained offset is equal to the consumer's minimum committed offset;

[0101] (2) Maximum: Data will only be deleted when the size of a small partition image reaches its configured maximum limit;

[0102] (3) Fixed offset range: When the offset range stored in the small partition image is greater than the configured maximum offset range, delete the data;

[0103] (4) Generate retention strategies based on the AI ​​intelligent strategies of the intelligent scheduler;

[0104] (c) The data synchronization module is used to obtain the data of the topic requested by the coordinator from the intermediary agent module and store the data in a small partition image. If there are no active consumers, the data synchronization module stops or continues to obtain data according to the configuration.

[0105] (d) A small partition mirroring module that internally stores data from the intermediary agent module, including file system, database, or in-memory storage.

[0106] At this point, the intelligent scheduler internally stores the execution of intelligent scheduling algorithms, which are used to generate intelligent policies and guide the coordinator module to work, so as to ensure a high message hit rate in the consumer agent while minimizing the amount of messages stored.

[0107] Therefore, the system can significantly reduce network bandwidth consumption without changing consumer behavior and logic, while also achieving lower latency and other performance characteristics. It also minimizes storage costs while maintaining a high message access hit rate, thus achieving multi-objective optimization of performance and cost.

[0108] On the other hand, the present invention also discloses an intelligent scheduler applied to the novel distributed messaging system in the above embodiments, which executes the following specific workflow through an intelligent scheduling algorithm:

[0109] S1. Data Collection and Feature Extraction: Collect and extract access pattern features for each consumer i from the coordinator, including: access frequency f i (t), access latency requirement δ i (t), the offset range of consumption [o i，min (t), o i，max Consumer attributes A, including [t], geographical location, and device type. i Historical Access Mode H i (t);

[0110] S2. Advanced Consumer Behavior Modeling: A Long Short-Term Memory (LSTM) network model is used to model the access frequency and consumption interval of each consumer i, capturing temporal dependencies and complex patterns to predict future access frequencies. and consumption range have:

[0111]

[0112] Among them, LSTM f and LSTM o These are LSTM models for predicting access frequency and consumption intervals, respectively.

[0113] S3. Collaborative Filtering and Preference Learning: Utilizing collaborative filtering techniques, based on the similarity between consumers, enhances the prediction of the probability of a specified offset being accessed. The specific operational process for enhancing the prediction of the probability of a specified offset being accessed includes:

[0114] S31. The similarity between consumers i and j is presumably measured by the similarity function sim(i, j);

[0115] S32. Calculate consumer similarity using cosine similarity:

[0116]

[0117] Among them, f i and f j It is the access frequency vector of consumers i and j;

[0118] S33. For consumers i and j, adjust their access probabilities. and To make preference P i For consumer o, enhancements include:

[0119]

[0120] Where the consumption range of i is j's consumption range is

[0121] S4. Multi-objective probability estimation: Combining access probability and storage cost, multi-objective optimization is performed. The specific operation process for multi-objective optimization includes:

[0122] S41. Set objectives: maximize hit rate H and minimize storage cost S;

[0123] S42. Combining the access probabilities of all consumers, calculate the total access probability. The total probability P(o, t) that a single offset consumer o is accessed at time t is:

[0124]

[0125] Where P i (o, t) represents the access probability of a single consumer;

[0126] S43. Assuming that the storage cost of each offset is the same, define the storage cost of each offset consuming o as C(o) = 1;

[0127] S44. Construct a weighted objective function as the objective function for multi-objective optimization:

[0128]

[0129] Where α and β are weighting parameters, used to consider the balanced hit rate and storage cost, respectively;

[0130] S5. Establish a reinforcement learning-driven optimization prefetching strategy: Use deep reinforcement learning (DRL), such as deep Q-networks (DQN), to dynamically learn and optimize the prefetching strategy. The usage of deep Q-network algorithms includes:

[0131] S51. Preset State S t Action A t and reward R t To strengthen the learning framework, in which:

[0132] (1) For state S t :

[0133] S t ={S(t-1), P(o,t), A i H i (t)}

[0134] State S t Includes: the offset set S(t-1) stored at the current time t, the predicted access probability P(o,t), and the consumer attribute A. i and historical access mode H i (t);

[0135] (2) For consumer attribute A t :

[0136] A + ={S(t)}

[0137] Consumer attribute A t Choose a new offset set S(t) for prefetching;

[0138] (3) Regarding the reward R t :

[0139]

[0140] Represents reward R t The reward function considers the hit rate and storage cost C(o), where λ and μ are weight parameters;

[0141] S52. Formula for updating the Q function using a deep Q-network:

[0142]

[0143] in:

[0144] Q(S t , A t ;θ) represents the current state S t Take action A t The Q value, with parameter θ;

[0145] α represents the learning rate, which is used to control the update step size;

[0146] R t This represents the immediate reward received at time point t;

[0147] γ represents the discount factor, used to weigh the importance of current rewards against future rewards;

[0148] This is represented as the next state S. t+1 The largest Q value among all possible actions;

[0149] S53. Strategy Selection: Employ ε-greedy strategies to balance exploration and exploitation.

[0150]

[0151] Where ∈ is a very small positive number;

[0152] S6. Introduction of Dynamic Adjustment and Adaptive Feedback Mechanism: An adaptive feedback mechanism is introduced to continuously monitor system performance and dynamically adjust model parameters. The specific calculation steps for the adaptive feedback mechanism to dynamically adjust model parameters include:

[0153] S61. Obtain the actual hit rate H of monitoring through real-time monitoring. actual (t) and storage usage S actual (t);

[0154] S62. Calculate the prediction error Error(t) and use optimization methods, including gradient descent, to adjust the prediction model parameters to reduce the error:

[0155]

[0156] Where P(o,t) represents the probability of a consumer's visit, and S(t) represents the offset set;

[0157] S63. Dynamically adjust the storage capacity C(t) based on system load and storage usage:

[0158]

[0159] Where C0 is the initial capacity, ΔC is the adjustment step size, and S...max This is the maximum storage capacity;

[0160] At this point, the intelligent scheduler, through continuous dynamic adjustment and feedback mechanisms, can adaptively respond to complex and changing consumer behavior patterns, ensuring that the system can maintain efficient operation in various scenarios.

[0161] In summary, by combining advanced deep learning models, collaborative filtering, multi-objective optimization, and reinforcement learning, the intelligent scheduler can dynamically and intelligently estimate and select the message offset range that needs to be prefetched. This not only improves the message access hit rate but also effectively controls storage costs. It has high adaptability and scalability, achieving a virtuous balance between minimizing storage costs and maximizing access hit rate.

[0162] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0163] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0164] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of the method executed by the intelligent scheduler in any of the novel distributed messaging systems described above.

[0165] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0166] This application also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus.

[0167] Memory, used to store computer programs;

[0168] When a processor executes a program stored in memory, it implements the steps of the method executed by the intelligent scheduler in the novel distributed messaging system described above.

[0169] The communication bus mentioned in the above-mentioned electronic devices can be a standard bus for interconnecting peripheral components or an extended industrial standard structure bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0170] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0171] The memory may include random access memory or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0172] The processors mentioned above can be general-purpose processors, including central processing units, network processors, etc.; they can also be digital signal processors, application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0173] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0174] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0175] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, in the embodiments of this invention, "multiple" refers to two or more. Moreover, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

Claims

1. A novel distributed messaging system, characterized in that, It consists of an intelligent scheduler and a consumer agent, which are installed in each distributed computing node. The consumer agent internally includes a data synchronization module, a small partition mirroring module, a mediator module, and a coordinator module, wherein: The intelligent scheduler internally stores and executes intelligent scheduling algorithms to generate intelligent policies and guide the coordinator module to ensure a high message hit rate in the consumer agent while minimizing the amount of messages stored. The intermediary / agent module acts as an agent, providing services to consumers. The coordinator module handles registrations from the intermediary agent module and has an internal smart policy to instruct the data synchronization module to obtain data from the intermediary agent module. The data synchronization module is used to obtain the data of the topic requested by the coordinator from the intermediary agent module and store the data in a small partition image. If there are no active consumers, the data synchronization module stops or continues to obtain data according to the configuration. A small partition mirroring module that internally stores data from the intermediary agent module, including file systems, databases, or in-memory storage.

2. The novel distributed messaging system as described in claim 1, characterized in that, When a consumer subscribes to a topic or submits an offset for a topic, the intermediary agent module sends the topic, consumer group, and requested or submitted offset data to the coordinator. When a consumer consumes data, the intermediary agent module is used to retrieve data from a small partitioned image and respond to the consumer.

3. The novel distributed messaging system as described in claim 1 or 2, characterized in that, The intelligent strategies within the coordinator module include: (1) Minimum: The minimum retained offset is equal to the consumer's minimum committed offset; (2) Maximum: Data will only be deleted when the size of a small partition image reaches its configured maximum limit; (3) Fixed offset range: When the offset range stored in the small partition image is greater than the configured maximum offset range, delete the data; (4) Generate retention strategies based on the AI ​​intelligent strategies of the intelligent scheduler.

4. An intelligent scheduler, characterized in that, Applied to the novel distributed messaging system described in any one of claims 1-3, the following specific workflow is executed through an intelligent scheduling algorithm: S1. Data Collection and Feature Extraction: Collect and extract access pattern features for each consumer i from the coordinator, including: access frequency f i (t), access latency requirement δ i (t), the offset range of consumption [o i,min (t),o i,max Consumer attributes A, including [t], geographical location, and device type. i Historical Access Mode H i (t); S2. Advanced Consumer Behavior Modeling: A Long Short-Term Memory (LSTM) network model is used to model the access frequency and consumption interval of each consumer i, capturing temporal dependencies and complex patterns to predict future access frequencies. and consumption range have: Among them, LSTM f and LSTM o These are LSTM models for predicting access frequency and consumption intervals, respectively. S3. Collaborative Filtering and Preference Learning: Utilizing collaborative filtering techniques, based on the similarity between consumers, to enhance the prediction of the probability of a specified offset being accessed; S4. Multi-objective probability estimation: Combining access probability and storage cost to perform multi-objective optimization; S5. Establish a reinforcement learning-driven optimization prefetching strategy: Use deep Q-network algorithm for deep reinforcement learning to dynamically learn and optimize the prefetching strategy; S6. Introduction of dynamic adjustment and adaptive feedback mechanism: An adaptive feedback mechanism is introduced to continuously monitor system performance and dynamically adjust model parameters.

5. The intelligent scheduler as described in claim 4, characterized in that, The specific operational procedures for enhancing the prediction of the probability of a specified offset being accessed in step S3 include: S31. The similarity between consumers i and j is presumably measured by the similarity function sim(i,j); S32. Calculate consumer similarity using cosine similarity: Among them, f i and f j It is the access frequency vector of consumers i and j; S33. For consumers i and j, adjust their access probabilities. and To make preference P i For consumer o, enhancements include: Where the consumption range of i is j's consumption range is 6. The intelligent scheduler as described in claim 4, characterized in that, The specific operational procedures for multi-objective optimization in step S4 include: S41. Set objectives: maximize hit rate H and minimize storage cost S; S42. Combining the access probabilities of all consumers, calculate the total access probability. The total probability P(o,t) that a single offset consumer o is accessed at time t is: Where P i (o,t) represents the access probability of a single consumer; S43. Assuming that the storage cost of each offset is the same, define the storage cost of each offset consuming o as C(o) = 1; S44. Construct a weighted objective function as the objective function for multi-objective optimization: Here, α and β are weighting parameters, used to consider the balanced hit rate and storage cost, respectively.

7. The intelligent scheduler as described in claim 4, characterized in that, The method of using the deep Q-network algorithm in step S5 includes: S51. Preset State S t Action A t and reward R t To strengthen the learning framework, in which: S t ={S(t-1),P(o,t),A i ,H i (t)} State S t Includes: the offset set S(t-1) stored at the current time t, the predicted access probability P(o,t), and consumer attribute A. i and historical access mode H i (t); A t ={S(t)} Consumer attribute A t Choose a new offset set S(t) for prefetching; Represents reward R t The reward function considers the hit rate and storage cost C(o), where λ and μ are weight parameters; S52. Formula for updating the Q function using a deep Q-network: in: Q(S t A t ;θ) represents the current state S t Take action A t The Q value, with parameter θ; α represents the learning rate, which is used to control the update step size; R t This represents the immediate reward received at time point t; γ represents the discount factor, used to weigh the importance of current rewards against future rewards; This is represented as the next state S. t+1 The largest Q value among all possible actions; S53. Employ ε-greedy strategies for action selection to balance exploration and exploitation: Where ∈ is a very small positive number.

8. The intelligent scheduler as described in claim 4, characterized in that, The specific computational steps for dynamically adjusting model parameters using the adaptive feedback mechanism in step S6 include: S61. Obtain the actual hit rate H of monitoring through real-time monitoring. actual (t) and storage usage S actual (t); S62. Calculate the prediction error Error(t) and use optimization methods, including gradient descent, to adjust the prediction model parameters to reduce the error: Where P(o,t) represents the probability of a consumer's visit, and S(t) represents the offset set; S63. Dynamically adjust the storage capacity C(t) based on system load and storage usage: Where C0 is the initial capacity, ΔC is the adjustment step size, and S... max That is the maximum storage capacity.

9. A computer-readable storage medium, characterized in that, The system stores a computer program that, when executed by a processor, causes the processor to perform the steps of the method executed by the intelligent scheduler in the novel distributed messaging system as described in any one of claims 1 to 3.

10. A computer device, characterized in that, The system includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method executed by the intelligent scheduler in the novel distributed messaging system as described in any one of claims 1 to 3.

Images