Novel distributed message system, intelligent scheduler, equipment and storage medium

Abstract

The invention discloses a novel distributed message system, which is composed of an intelligent scheduler and a consumer agent, the intelligent scheduler and the consumer agent are arranged in each distributed computing node, and the intelligent scheduler is internally stored with an intelligent scheduling execution algorithm for generating an intelligent strategy, therefore, a high message hit rate in the consumer agency is ensured, and the stored message quantity is minimum at the same time; the consumer agent is internally provided with an intermediate agent module used for serving as an agent; the coordinator module is used for indicating the data synchronization module to acquire the data; the data synchronization module is used for acquiring data according to the request of the coordinator and storing the data into the small partition mirror image; and data are stored in the small partition mirror image module. According to the method, the application can receive the message from the consumer agent in the node where the application is located, message transmission from the agent to the node where the application runs is reduced, and due to the fact that most of message consumption occurs in one node, message transmission on the network is greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of message processing, and in particular to a novel distributed message system, an intelligent scheduler, a device and a storage medium. Background Art

[0002] As a way to intelligently integrate the real and virtual worlds, IoT technology has developed rapidly in recent years. By 2050, it is expected that there will be 24 billion interconnected devices in the network, which will generate a large number of messages and events for communication. Messaging systems play an important role in data transmission in the IoT era.

[0003] The most popular messaging system at present is the publish-subscribe messaging system, such as Kafka or Pulsar, in which producers send data to brokers and consumers receive data from brokers, where each consumer establishes a connection with the broker to consume data.

[0004] Taking Kafka as an example, in the existing publish-subscribe messaging system implementation, consumers in each application will establish their own independent connection with the broker to consume data. If multiple applications want to consume the same topic, each consumer will independently obtain data from the partition. Even if the consumers are located on the same node, the same data will be sent to each consumer, which means that the same data will be sent from the same broker to the same node multiple times. Considering that many applications are connected to the broker and consume data from the same topic, this sending mechanism may cause message storms, reduce network performance, and in the worst case, may cause network outages, and also result in a lot of additional network bandwidth requirements and cost consumption.

[0005] To this end, this application specifically proposes a new distributed messaging system to solve the above technical problems. Summary of the invention

[0006] The main purpose of the present invention is to provide a new type of distributed message system, which introduces a consumer agent in the node and uses appropriate algorithms and mechanisms to ensure that the messages in the agent are transmitted to the node only once as much as possible, so that the application can receive messages from the consumer agent in the node where it is located, so as to solve the technical problems raised in the background technology and reduce the message transmission from the agent to the node where the application is running.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] A novel distributed message system is composed of an intelligent scheduler and a consumer agent. The intelligent scheduler and the consumer agent are arranged in each distributed computing node. The consumer agent is internally provided with a data synchronization module, a small partition mirror module, an intermediary agent module and a coordinator module, wherein:

[0009] The intelligent scheduler has an internal storage that executes the intelligent scheduling algorithm, which is used to generate intelligent strategies and guide the work of the coordinator module to ensure a high message hit rate in the consumer agent while minimizing the amount of messages stored;

[0010] The intermediary agent module is used to act as an agent and provide services to consumers;

[0011] The coordinator module is used to process the registration from the intermediary agent module, and has an intelligent strategy set inside to instruct the data synchronization module to obtain data from the intermediary agent module;

[0012] A data synchronization module is used to obtain data of the topic requested by the coordinator from the intermediary agent module and store the data in a small partition mirror. If there is no active consumer, the data synchronization module stops or continues to obtain data according to the configuration;

[0013] A small partition mirror module that internally stores data from the intermediate proxy module, including file system, database or in-memory storage.

[0014] Preferably, when a consumer subscribes to a topic or submits an offset of a topic in the intermediary proxy module, the intermediary proxy module is used to send the topic, consumer group and requested or submitted offset data to the coordinator.

[0015] Preferably, when a consumer consumes data in the intermediate proxy module, the intermediate proxy module is used to obtain data from the small partition image and respond to the consumer.

[0016] Preferably, the internal intelligent strategy of the coordinator module includes:

[0017] (1) Minimum: The minimum retained offset is equal to the consumer's minimum committed offset;

[0018] (2) Maximum: Data is deleted only when the size of a small partition image reaches its configured maximum limit;

[0019] (3) Fixed offset range: When the offset range stored in the small partition image is larger than the configured maximum offset range, the data is deleted;

[0020] (4) Generate a retention strategy based on the AI ​​intelligent strategy of the intelligent scheduler.

[0021] An intelligent scheduler is applied to any of the novel distributed message systems described above, and executes the following specific workflows through an intelligent scheduling algorithm:

[0022] S1. Data collection and feature extraction: Collect and extract access pattern features of each consumer i from the coordinator, including: access frequency f i (t), access delay requirement δ i (t), consumption offset interval [o i，min (t), o i，max (t)], consumer attributes including geographic location and device type A i 、Historical access mode H i (t);

[0023] S2. Advanced consumer behavior modeling: The long short-term memory network (LSTM) model is used to model the access frequency and consumption interval of each consumer i to capture temporal dependencies and complex patterns, thereby predicting future access frequencies. and consumption range have:

[0024]

[0025] Among them, LSTM f and LSTM o They are the LSTM model for predicting the access frequency and the LSTM model for predicting the consumption range.

[0026] S3. Collaborative filtering and preference learning: Using collaborative filtering technology, based on the similarities between consumers, the probability prediction of a specified offset being accessed is enhanced;

[0027] S4. Multi-objective probability estimation: Combine access probability and storage cost to perform multi-objective optimization;

[0028] S5. Establish a reinforcement learning driven optimized prefetching strategy: Use deep Q-network algorithm for deep reinforcement learning to dynamically learn and optimize the prefetching strategy;

[0029] S6. Introduce dynamic adjustment and adaptive feedback mechanism: Introduce adaptive feedback mechanism to continuously monitor system performance and dynamically adjust model parameters

[0030] Preferably, the specific operation flow of enhancing the probability prediction of the designated offset being accessed in step S3 includes:

[0031] S31. The similarity between consumers i and j is measured by the similarity function sim(i, j);

[0032] S32. Use cosine similarity to calculate consumer similarity:

[0033]

[0034] Among them, f i and f j is the access frequency vector of consumers i and j;

[0035] S33. For consumers i and j, adjust their access probability and To make the preference P i Enhanced, for consumption o, there are:

[0036]

[0037] The consumption range of i is The consumption range of j is

[0038] Preferably, the specific operation process of performing multi-objective optimization in step S4 includes:

[0039] S41. Set the goal: maximize the hit rate H and minimize the storage cost S;

[0040] S42. The total access probability is calculated by combining the access probabilities of all consumers. The total probability P(o, t) that a single offset consumer o is accessed at time t is:

[0041]

[0042] Where P i (o, t) is the access probability of a single consumer;

[0043] S43. Assuming that the storage cost of each offset is the same, define the storage cost of each offset consumption o as C(o) = 1;

[0044] S44. Construct a weighted objective function as the multi-objective optimization objective function:

[0045]

[0046] Among them, α and β are weight parameters, which are used to balance the hit rate and storage cost respectively.

[0047] Preferably, the method for using the deep Q network algorithm in step S5 includes:

[0048] S51. Default state S t 、Action A t and reward R t , in a reinforcement learning framework, where:

[0049] S t={S(t-1), P(o, t), A i , H i (t)}

[0050] Indicates state S t Contains: the offset set S(t-1) stored at the current time t, the predicted access probability P(o, t), and the consumer attribute A i and historical access pattern H i (t);

[0051] A t ={S(t)}

[0052] Represents consumer attribute A t Select a new offset set S(t) for prefetching;

[0053]

[0054] Reward R t The reward function takes into account the hit rate and storage cost C(o), where λ and μ are weight parameters;

[0055] S52. Formula for updating the Q function using a deep Q network:

[0056]

[0057] in:

[0058] Q(S t , A t ; θ) represents the current state S t Next take action A t The Q value of , the parameter is θ;

[0059] α is the learning rate, which is used to control the update step size;

[0060] R t Represented as the immediate reward obtained at time point t;

[0061] γ is a discount factor that weighs the importance of current rewards and future rewards;

[0062] Indicated as in the next state S t+1 Next, the maximum Q value among all possible actions;

[0063] S53. Use ε-greedy to select actions to balance exploration and exploitation:

[0064]

[0065] Where ∈ is a very small positive number.

[0066] Preferably, the specific operation steps of the adaptive feedback mechanism in step S6 to dynamically adjust the model parameters include:

[0067] S61. Obtaining the actual hit rate H through real-time monitoring actual (t) and storage usage S actual (t);

[0068] S62. Calculate the prediction error Error(t) and use optimization methods including gradient descent to adjust the prediction model parameters to reduce the error:

[0069]

[0070] Where P(o, t) represents the consumer's access probability, and S(t) represents the offset set;

[0071] S63. Dynamically adjust the storage capacity C(t) according to the system load and storage usage:

[0072]

[0073] Where C0 is the initial capacity, ΔC is the adjustment step size, S max is the maximum storage capacity.

[0074] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method executed by the intelligent scheduler in the above-mentioned novel distributed messaging system.

[0075] On the other hand, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method executed by the intelligent scheduler in the above-mentioned new distributed messaging system.

[0076] It can be seen from the above technical solution that the present invention provides a new distributed message system. Compared with the prior art, the present invention has the following advantages:

[0077] 1. The present invention constructs a distributed messaging system by setting up intelligent schedulers and consumer agents at distributed computing nodes. It can achieve an overall upgrade of messaging service functions while seamlessly integrating the existing distributed messaging system, thereby reducing message transmission from the agent to the node where the application is running.

[0078] 2. The present invention introduces consumer agents into nodes, generates intelligent strategies based on intelligent scheduling algorithms and guides the operation of the coordinator module, so that messages in the agent are transmitted to the node only once as much as possible, so that the application can receive messages from the consumer agent in the node where it is located. At the same time, using this method, most of the message consumption occurs within one node, and the message transmission on the network will be greatly reduced.

[0079] 3. The present invention can significantly reduce network bandwidth consumption without changing consumer behavior and logic, while having lower latency and other performance, and achieve a high message access hit rate while minimizing storage costs, thereby achieving multi-objective optimization of performance and cost.

[0080] 4. The intelligent scheduler of the present invention combines advanced deep learning models, collaborative filtering, multi-objective optimization and reinforcement learning to dynamically and intelligently estimate and select the message offset interval that needs to be pre-fetched, which not only improves the hit rate of message access, but also effectively controls the storage cost. It has high adaptability and scalability, and achieves a benign balance between minimizing storage costs and maximizing access hit rates.

[0081] 5. The intelligent scheduler of the present invention can adaptively respond to complex and changing consumer behavior patterns through continuous dynamic adjustment and feedback mechanisms, ensuring that the system can maintain efficient operation in various scenarios.

[0082] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become easy to understand through the following description. Of course, it is not necessary to achieve all of the advantages described above simultaneously for any product implementing the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The drawings constituting a part of the present application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0084] Figure 1 It is a schematic diagram of the overall module architecture of the system of the present invention;

[0085] Figure 2 It is a schematic diagram of the structural response of the consumer agent and the intelligent scheduler of the present invention;

[0086] Figure 3 It is a schematic diagram of the structural framework of the consumer agent of the present invention;

[0087] Figure 4 The figure is a schematic diagram of the framework structure of the Kafka system in the prior art. DETAILED DESCRIPTION

[0088] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. In the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0089] In the embodiment, see Figures 1 to 4 .

[0090] In the prior art, the most popular messaging system is a publish-subscribe messaging system, such as Kafka or Pulsar. Figure 4 As shown, Kafka is taken as an example, but the present invention is not limited to Kafka. In the existing publish-subscribe messaging system implementation, consumers in each application will establish their own independent connection with the broker to consume data.

[0091] like Figure 4 As shown in the figure, if there are multiple applications consuming the same topic, each consumer will independently obtain data from the partition, and the same data will be sent to each consumer even if the consumers are located on the same node. This means that the same data will be sent multiple times from the same broker to the same node. Considering that many applications are connected to the broker and consume data from the same topic, this sending mechanism may cause message storms, degrade network performance, and in the worst case, may cause network outages, and also result in a lot of additional network bandwidth requirements and cost consumption.

[0092] Therefore, Figure 1 , Figure 2 and Figure 3 As shown, a new type of distributed message system proposed in an embodiment of the present invention is composed of an intelligent scheduler and a consumer agent, and the intelligent scheduler and the consumer agent are arranged in each distributed computing node. It can achieve an overall upgrade of the message service function while seamlessly integrating the existing distributed message system, so as to reduce the message transmission from the agent to the node where the application is running.

[0093] By introducing consumer agents in the nodes and using appropriate algorithms and mechanisms to ensure that messages in the agents are transmitted to the nodes only once, applications can receive messages from the consumer agents in their nodes. With this approach, most message consumption occurs within one node, and message transmission on the network is greatly reduced.

[0094] Further, such as Figure 2 As shown, a consumer proxy and an intelligent scheduler are created in each distributed computing node. For each application's consumers, they connect to and consume data from the consumer proxy instead of consuming data directly from the broker. Under the guidance of the intelligent scheduler, the consumer proxy synchronizes data from the broker on behalf of the consumer and distributes messages to each application's consumer. The intelligent scheduler needs to ensure the maximum message access hit rate while storing as few messages as possible.

[0095] From the application's perspective, it consumes data from the local host rather than from the proxy. The behavior and interface provided by the consumer proxy should be the same as the proxy, so that the application is not affected. At the same time, it obtains data from the local host, and its latency and other indicators will be smaller.

[0096] From the perspective of the broker, the consumer agent consumes data from it as an ordinary consumer. The interface between the broker and the consumer agent remains the same as the interface between the traditional broker and the consumer, and the interface between the consumer agent and the consumer also remains the same as the interface between the traditional broker and the consumer.

[0097] The system is as Figure 3 As shown, the consumer agent is internally provided with a data synchronization module, a small partition mirror module, an intermediary agent module and a coordinator module, specifically including:

[0098] (a) An intermediary agent module is used to act as an agent to provide services to consumers. When a consumer subscribes to a topic or submits an offset to a topic, the intermediary agent module is used to send the topic, consumer group, and requested or submitted offset data to the coordinator; when a consumer consumes data, the intermediary agent module is used to obtain data from the small partition mirror and respond to the consumer;

[0099] (b) A coordinator module is used to process registrations from the intermediary agent module and has an internal intelligent strategy for instructing the data synchronization module to obtain data from the intermediary agent module, wherein the internal intelligent strategy of the coordinator module includes:

[0100] (1) Minimum: The minimum retained offset is equal to the consumer's minimum committed offset;

[0101] (2) Maximum: Data is deleted only when the size of a small partition image reaches its configured maximum limit;

[0102] (3) Fixed offset range: When the offset range stored in the small partition image is larger than the configured maximum offset range, the data is deleted;

[0103] (4) Generate a retention strategy based on the AI ​​intelligent strategy of the intelligent scheduler;

[0104] (c) a data synchronization module, used to obtain data of the topic requested by the coordinator from the intermediary agent module and store the data in the small partition mirror, and if there is no active consumer, the data synchronization module stops or continues to obtain data according to the configuration;

[0105] (d) A small partition mirror module that internally stores data from the intermediate proxy module, including storage in a file system, a database, or memory.

[0106] At this time, the intelligent scheduler has an internal storage that executes the intelligent scheduling algorithm, which is used to generate intelligent strategies and guide the work of the coordinator module to ensure a high message hit rate in the consumer agent while minimizing the amount of messages stored.

[0107] Therefore, the system can significantly reduce network bandwidth consumption without changing consumer behavior and logic, while having lower latency and other performance, and achieve high message access hit rate while minimizing storage costs, thereby achieving multi-objective optimization of performance and cost.

[0108] On the other hand, the present invention also discloses an intelligent scheduler, which is applied to the novel distributed message system in the above embodiment and executes the following specific workflow through an intelligent scheduling algorithm:

[0109] S1. Data collection and feature extraction: Collect and extract access pattern features of each consumer i from the coordinator, including: access frequency f i (t), access delay requirement δ i (t), consumption offset interval [o i，min (t), o i，max (t)], consumer attributes including geographic location and device type A i 、Historical access mode H i (t);

[0110] S2. Advanced consumer behavior modeling: The long short-term memory network (LSTM) model is used to model the access frequency and consumption interval of each consumer i to capture temporal dependencies and complex patterns, thereby predicting future access frequencies. and consumption range have:

[0111]

[0112] Among them, LSTM f and LSTM o They are the LSTM model for predicting the access frequency and the LSTM model for predicting the consumption range.

[0113] S3. Collaborative filtering and preference learning: Collaborative filtering technology is used to enhance the probability prediction of a specified offset being accessed based on the similarity between consumers. The specific operation process of enhancing the probability prediction of a specified offset being accessed includes:

[0114] S31. The similarity between consumers i and j is measured by the similarity function sim(i, j);

[0115] S32. Use cosine similarity to calculate consumer similarity:

[0116]

[0117] Among them, f i and f j is the access frequency vector of consumers i and j;

[0118] S33. For consumers i and j, adjust their access probability and To make the preference P i Enhanced, for consumption o, there are:

[0119]

[0120] The consumption range of i is The consumption range of j is

[0121] S4. Multi-objective probability estimation: Combine access probability and storage cost to perform multi-objective optimization. The specific operation process of multi-objective optimization includes:

[0122] S41. Set the goal: maximize the hit rate H and minimize the storage cost S;

[0123] S42. The total access probability is calculated by combining the access probabilities of all consumers. The total probability P(o, t) that a single offset consumer o is accessed at time t is:

[0124]

[0125] Where P i (o, t) is the access probability of a single consumer;

[0126] S43. Assuming that the storage cost of each offset is the same, define the storage cost of each offset consumption o as C(o) = 1;

[0127] S44. Construct a weighted objective function as the multi-objective optimization objective function:

[0128]

[0129] Among them, α and β are weight parameters, which are used to consider the balance between hit rate and storage cost respectively;

[0130] S5. Establish a reinforcement learning-driven optimized prefetching strategy: Use deep reinforcement learning (DRL), such as the Deep Q Network (DQN), to dynamically learn and optimize the prefetching strategy. The use of the Deep Q Network algorithm includes:

[0131] S51. Default state S t 、Action A t and reward R t , in a reinforcement learning framework, where:

[0132] (1) For state S t :

[0133] S t ={S(t-1), P(o, t), A i , H i (t)}

[0134] Indicates state S t Contains: the offset set S(t-1) stored at the current time t, the predicted access probability P(o, t), and the consumer attribute A i and historical access pattern H i (t);

[0135] (2) For consumer attribute A t :

[0136] A + ={S(t)}

[0137] Represents consumer attribute A t Select a new offset set S(t) for prefetching;

[0138] (3) For reward R t :

[0139]

[0140] Reward R t The reward function takes into account the hit rate and storage cost C(o), where λ and μ are weight parameters;

[0141] S52. Formula for updating the Q function using a deep Q network:

[0142]

[0143] in:

[0144] Q(S t , A t ; θ) represents the current state S t Next take action A t The Q value of , the parameter is θ;

[0145] α is the learning rate, which is used to control the update step size;

[0146] R t Represented as the immediate reward obtained at time point t;

[0147] γ is a discount factor that weighs the importance of current rewards and future rewards;

[0148] Indicated as in the next state S t+1 Next, the maximum Q value among all possible actions;

[0149] S53. Strategy selection: Use ε-greedy to perform strategy selection actions to balance exploration and exploitation:

[0150]

[0151] Where ∈ is a very small positive number;

[0152] S6. Introduce dynamic adjustment and adaptive feedback mechanism: Introduce an adaptive feedback mechanism to continuously monitor system performance and dynamically adjust model parameters. The specific operation steps of the adaptive feedback mechanism to dynamically adjust model parameters include:

[0153] S61. Obtaining the actual hit rate H through real-time monitoring actual (t) and storage usage S actual (t);

[0154] S62. Calculate the prediction error Error(t) and use optimization methods including gradient descent to adjust the prediction model parameters to reduce the error:

[0155]

[0156] Where P(o, t) represents the consumer's access probability, and S(t) represents the offset set;

[0157] S63. Dynamically adjust the storage capacity C(t) according to the system load and storage usage:

[0158]

[0159] Where C0 is the initial capacity, ΔC is the adjustment step size, Smax is the maximum storage capacity;

[0160] At this time, the intelligent scheduler can adaptively respond to complex and changing consumer behavior patterns through continuous dynamic adjustment and feedback mechanisms, ensuring that the system can maintain efficient operation in various scenarios.

[0161] In summary, the intelligent scheduler combines advanced deep learning models, collaborative filtering, multi-objective optimization and reinforcement learning to dynamically and intelligently estimate and select the message offset intervals that need to be prefetched. This not only improves the hit rate of message access, but also effectively controls storage costs. It has high adaptability and scalability, and achieves a benign balance between minimizing storage costs and maximizing access hit rates.

[0162] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0163] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0164] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of the method executed by the intelligent scheduler in any of the novel distributed message systems in the above embodiments.

[0165] It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above method.

[0166] The embodiment of the present application also provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus.

[0167] Memory, used to store computer programs;

[0168] The processor is used to implement the steps of the method executed by the intelligent scheduler in the above-mentioned new distributed message system when executing the program stored in the memory.

[0169] The communication bus mentioned in the above electronic device can be a peripheral component interconnect standard bus or an extended industrial standard architecture bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0170] The communication interface is used for communication between the above electronic device and other devices.

[0171] The memory may include a random access memory, or may include a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0172] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor, etc.; it can also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0173] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0174] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0175] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the meaning of "and / or" appearing in the full text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or schemes that A and B meet at the same time. In addition, in the embodiments of the present invention, "multiple" refers to more than two. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A new distributed messaging system, characterized in that: It consists of an intelligent scheduler and a consumer agent, which are set in each distributed computing node. The consumer agent is internally provided with a data synchronization module, a small partition mirror module, an intermediary agent module and a coordinator module, wherein: The intelligent scheduler has an internal storage that executes the intelligent scheduling algorithm, which is used to generate intelligent strategies and guide the work of the coordinator module to ensure a high message hit rate in the consumer agent while minimizing the amount of messages stored; The intermediary agent module is used to act as an agent and provide services to consumers; The coordinator module is used to process the registration from the intermediary agent module, and has an intelligent strategy set inside to instruct the data synchronization module to obtain data from the intermediary agent module; A data synchronization module is used to obtain data of the topic requested by the coordinator from the intermediary agent module and store the data in a small partition mirror. If there is no active consumer, the data synchronization module stops or continues to obtain data according to the configuration; A small partition mirror module that internally stores data from the intermediate proxy module, including file system, database or in-memory storage.

2. The novel distributed messaging system as claimed in claim 1, characterized in that: When a consumer subscribes to a topic or submits an offset of a topic in the intermediary proxy module, the intermediary proxy module is used to send the topic, consumer group and requested or submitted offset data to the coordinator; When the consumer consumes data, the intermediary agent module is used to obtain data from the small partition mirror and respond to the consumer.

3. The novel distributed messaging system according to claim 1 or 2, characterized in that: The internal intelligent strategy of the coordinator module includes: (1) Minimum: The minimum retained offset is equal to the consumer's minimum committed offset; (2) Maximum: Data is deleted only when the size of a small partition image reaches its configured maximum limit; (3) Fixed offset range: When the offset range stored in the small partition image is larger than the configured maximum offset range, the data is deleted; (4) Generate a retention strategy based on the AI ​​intelligent strategy of the intelligent scheduler.

4. An intelligent scheduler, characterized in that: The novel distributed messaging system according to any one of claims 1 to 3 above is applied to execute the following specific workflow through an intelligent scheduling algorithm: S1. Data collection and feature extraction: Collect and extract access pattern features of each consumer i from the coordinator, including: access frequency f i (t), access delay requirement δ i (t), consumption offset interval [o i,min (t),o i,max (t)], consumer attributes Ai including geographic location and device type, and historical access patterns Hi(t); S2. Advanced consumer behavior modeling: The long short-term memory network (LSTM) model is used to model the access frequency and consumption interval of each consumer i to capture temporal dependencies and complex patterns, thereby predicting the future access frequency f^ i (t) and consumption interval [o^ i，min (t), o^ i，max (t)], we have: f^ i (t)=LSTM f (H i (t)) [o^ i,min (t),o^ i,max (t)]=LSTM o (H i (t)) Among them, LSTM f and LSTM o They are the LSTM model for predicting the access frequency and the LSTM model for predicting the consumption range. S3. Collaborative filtering and preference learning: Using collaborative filtering technology, based on the similarities between consumers, the probability prediction of a specified offset being accessed is enhanced; S4. Multi-objective probability estimation: Combine access probability and storage cost to perform multi-objective optimization; S5. Establish a reinforcement learning driven optimized prefetching strategy: Use deep Q-network algorithm for deep reinforcement learning to dynamically learn and optimize the prefetching strategy; S6. Introduce dynamic adjustment and adaptive feedback mechanism: Introduce an adaptive feedback mechanism to continuously monitor system performance and dynamically adjust model parameters.

5. The novel distributed messaging system as claimed in claim 4, characterized in that: The specific operation flow of enhancing the probability prediction of the specified offset being accessed in step S3 includes: S31. The similarity between consumers i and j is measured by the similarity function sim(i,j); S32. Use cosine similarity to calculate consumer similarity: Among them, f i and f j is the access frequency vector of consumers i and j; S33. For consumers i and j, adjust their access probability f^ i (t) and f^ j (t), so that the preference P i Enhanced, for consumption o, there are: The consumption range of i is [o^i ,min (t),o^ i,max (t)], j's consumption interval is [o^ j,min (t),o^ j,max (t)].

6. The novel distributed messaging system as claimed in claim 4, characterized in that: The specific operation process of performing multi-objective optimization in step S4 includes: S41. Set the goal: maximize the hit rate H and minimize the storage cost S; S42. Combine the access probabilities of all consumers and calculate the total access probability. The total probability P(o,t) that a single offset consumer o is accessed at time t is: Where P i (o,t) is the access probability of a single consumer; S43. Assuming that the storage cost of each offset is the same, define the storage cost of each offset consumption o as C(o) = 1; S44. Construct a weighted objective function as the multi-objective optimization objective function: Among them, α and β are weight parameters, which are used to balance the hit rate and storage cost respectively.

7. The novel distributed messaging system as claimed in claim 4, characterized in that: The method for using the deep Q network algorithm in step S5 includes: S51. Default state S t 、Action A t and reward R t , in a reinforcement learning framework, where: S t ={S(t-1),P(o,t),A i ,H i (t)} Indicates state S t Contains: the offset set S(t-1) stored at the current time t, the predicted access probability P(o,t), and the consumer attribute A i and historical access pattern H i (t); A t ={S(t)} Represents consumer attribute A t Select a new offset set S(t) for prefetching; Reward R t The reward function takes into account the hit rate and storage cost C(o), where λ and μ are weight parameters; S52. Formula for updating the Q function using a deep Q network: in: Q(S t ,A t ; θ) represents the current state S t Next take action A t The Q value of , the parameter is θ; α is the learning rate, which is used to control the update step size; R t Represented as the immediate reward obtained at time point t; γ is a discount factor that weighs the importance of current rewards and future rewards; Indicated as in the next state S t+1 Next, the maximum Q value among all possible actions; S53. Use ε-greedy to select actions to balance exploration and exploitation: Where ∈ is a very small positive number.

8. The novel distributed messaging system as claimed in claim 4, characterized in that: The specific operation steps of the adaptive feedback mechanism in step S6 to dynamically adjust the model parameters include: S61. Obtaining the actual hit rate H through real-time monitoring actual (t) and storage usage S actual (t); S62. Calculate the prediction error Error(t) and use optimization methods including gradient descent to adjust the prediction model parameters to reduce the error: Where P(o,t) represents the consumer's access probability, and S(t) represents the offset set; S63. Dynamically adjust storage capacity C(t) according to system load and storage usage: Where C0 is the initial capacity, ΔC is the adjustment step size, S max is the maximum storage capacity.

9. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor executes the steps of the method executed by the intelligent scheduler in the new distributed messaging system as claimed in any one of claims 1 to 3.

10. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method executed by the intelligent scheduler in the new distributed message system as claimed in any one of claims 1 to 3.

Images