A dual-queue-based full-link decoupling asynchronous processing system
By using a Rust-based dual-queue decoupled asynchronous processing system, the problems of component coupling and inconsistent interfaces in traditional asynchronous data processing systems are solved, achieving plug-and-play functionality, fault isolation, and performance improvement for components.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHONGKE XINGTU MEASUREMENT & CONTROL TECH CO LTD
- Filing Date
- 2026-03-19
- Publication Date
- 2026-06-19
AI Technical Summary
In traditional asynchronous data processing systems, the production, consumption, and storage processes are strongly coupled, resulting in tight component dependencies, inconsistent interfaces, and complex collaboration logic. This makes it difficult to achieve flexible component replacement and rapid system deployment, and the asynchronous advantages of Rust are not fully utilized.
A Rust-based dual-queue decoupled asynchronous processing system is adopted, including a standardized interface layer, a dual-queue message relay layer, and a replaceable component layer. The interface is defined through Rust's trait feature, and two independent queues are implemented using tokio_mpmc::Queue, which shields direct interaction between components and achieves end-to-end decoupling and plug-and-play functionality.
It achieves no direct dependencies across the entire chain, component replacement requires no modification to other modules, adaptive speed adjustment, strong fault isolation capability, high development efficiency, significant performance advantages, flexible scalability, and improved reliability.
Smart Images
Figure CN122240170A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer software technology, and more specifically to a dual-queue, end-to-end decoupled asynchronous processing system. Background Technology
[0002] In traditional asynchronous data processing systems, the production, consumption, and storage of data are often strongly coupled, and there is a lack of a unified decoupling mechanism for cross-component interactions.
[0003] (1) Tight component dependencies: The code of the producer, consumer and storage modules are directly related (such as the producer hard-coded to call the consumer interface, and the consumer directly depends on a specific storage engine), which means that multiple module codes need to be modified when replacing a certain link, resulting in poor scalability.
[0004] (2) Inconsistent interfaces: Producers / consumers with different functions need to repeatedly develop communication logic with other modules (such as message format conversion and error handling), which results in low development efficiency and is prone to compatibility issues due to interface differences.
[0005] (3) Complex coordination logic: Asynchronous coordination of production, consumption and storage needs to be implemented manually (such as message buffering and retry mechanism); slow processing (such as database congestion) or failure (such as system crash) of a certain unit in the message passing link will affect the throughput of the entire link. These coordination logics increase the complexity of system design and make it difficult to guarantee reliability (such as message loss and processing blockage).
[0006] While existing technologies include message queues (such as RabbitMQ and Kafka) for decoupling communication, they lack standardized component interface specifications. Users still need to manually write adaptation code for components and queues, making it impossible to achieve "plug and play". At the same time, most systems do not have a full-link layered design of production-consumption-storage, making it difficult to support flexible component replacement and rapid system deployment.
[0007] Furthermore, in asynchronous scenarios, the characteristics of different programming languages (such as concurrency control and memory management) can affect the performance and reliability of the system. Rust's asynchronous runtime (such as tokio) combined with MPMC queues (such as tokio_mpmc::Queue) can provide efficient concurrent scheduling capabilities. However, existing technologies have not designed standardized three-layer decoupled systems based on these characteristics, making it difficult to fully leverage Rust's asynchronous advantages. Summary of the Invention
[0008] Based on the above analysis, this invention presents a fully decoupled asynchronous processing system based on dual queues, the specific implementation of which is as follows: it includes a standardized interface layer, a dual-queue message relay layer, and a replaceable component layer:
[0009] The standardized interface layer defines multiple core interfaces based on Rust's trait features, standardizes the behavior of components, and ensures that the components can be seamlessly connected to the dual-queue message relay layer. The component refers to a module that implements a certain standard trait and can be replaced or combined to complete a certain responsibility.
[0010] The dual-queue message relay layer is connected to the standardized interface layer. The dual-queue message relay layer is implemented based on Rust's tokio_mpmc::Queue (multi-producer-multi-consumer asynchronous queue) and contains two independent queues, each of which undertakes different relay tasks.
[0011] The replaceable component layer is connected to the dual-queue message relay layer. The replaceable component layer contains the components that implement the standardized interface layer. All the components interact indirectly through the independent queue and have no direct dependencies.
[0012] The three layers depend only on abstraction and queues and not on each other, thereby achieving full-link decoupling and allowing any of the components to be plug-and-play.
[0013] Preferably, the multiple core interfaces include a producer interface, which contains an abstract asynchronous method for message generation, stipulating that the producer needs to implement the message generation logic without needing to concern itself with the message transmission method.
[0014] Preferably, the multiple sets of core interfaces also include a consumer interface, which contains an abstract asynchronous method for message processing, stipulating that the consumer needs to implement the business processing logic of the message without needing to care about the source of the message.
[0015] Preferably, the multiple sets of core interfaces further include a storage interface, which contains an abstract asynchronous method for data persistence, stipulating that the storage component must implement data persistence logic without needing to concern itself with the data source.
[0016] Preferably, the independent queue includes a producer-consumer queue, which is responsible for receiving and temporarily storing messages output by the producer interface, and distributing them to the consumer interface according to the FIFO (first-in, first-out) strategy, thereby shielding the direct interaction between the producer and the consumer and balancing the production and consumption rates.
[0017] Preferably, the independent queue further includes a consumer-storage queue, which is responsible for receiving and temporarily storing the processing results output by the consumer interface, distributing them to the storage interface according to a FIFO strategy, shielding the direct interaction between the consumer and the storage, and balancing the consumption and storage rates.
[0018] Preferably, the component includes a producer component, which accesses the system by implementing the Producer trait. The producer component is used to generate messages and deliver the messages to the producer-consumer queue.
[0019] Preferably, the component further includes a consumer component, which accesses the system by implementing the Consumer trait, processes messages, and delivers the processing results to the consumer-storage queue.
[0020] Preferably, the component further includes a storage component, which accesses the system by implementing the Storage trait, obtains data to be stored from the consumption-storage queue, and is used for data persistence.
[0021] Preferably, it also includes a system coordinator, which is responsible for binding the component to the independent queue and starting the corresponding asynchronous task to drive the message to flow automatically throughout the entire link. Users only need to develop components that conform to the trait specification, and they can automatically access the system through the coordinator without having to worry about the details of asynchronous communication or asynchronous scheduling of the queue.
[0022] The beneficial effects of this invention are as follows:
[0023] (1) No direct dependencies across the entire chain: The “production-consumption” and “consumption-storage” links are decoupled by dual queues, and there are no direct calls between components. Replacing any component does not require modification of other modules.
[0024] (2) Rate adaptive adjustment: The two queues can independently absorb the rate difference between production-consumption and consumption-storage, avoiding full-link lag.
[0025] (3) Strong fault isolation capability: The failure of the storage component only affects the consumer-storage queue. Producers and consumers can continue to work normally. After the fault is recovered, the transmission will resume automatically and there will be no message loss.
[0026] (4) High development efficiency: Users only need to implement the core logic of standardized traits, without having to repeatedly develop general logic such as asynchronous scheduling and queue adaptation, which greatly reduces the development threshold.
[0027] (5) Significant performance advantages: Based on Rust's memory safety features and the efficient scheduling of the tokio runtime, combined with the lightweight asynchronous communication capabilities of the tokio_mpmc queue, low latency and high throughput can be maintained in high-concurrency scenarios.
[0028] (6) Flexible scalability: A certain link can be expanded independently (e.g., when there is a storage bottleneck, multiple storage component instances can be added to consume messages from the storage queue to achieve parallel storage).
[0029] (7) Improved reliability: The message relay layer has built-in asynchronous buffer and concurrency control to avoid blocking or message loss caused by mismatch between production / consumption rates, and the standardized interface unifies the error handling logic. Attached Figure Description
[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0031] Figure 1 This is a schematic diagram of the three-layer system in this invention;
[0032] Figure 2 This is a message flow timing diagram in this invention. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0034] like Figure 1 and Figure 2 As shown, this invention provides a fully decoupled asynchronous processing system based on dual queues. Its core is to divide the asynchronous processing system into a standardized interface layer, a dual-queue message relay layer, and a replaceable component layer. These three layers work collaboratively, achieving full-link decoupling and plug-and-play components through predefined interfaces and dual-queue relay. The specific system is as follows (implemented in Rust, utilizing its trait features to define interfaces, and relying on the tokio asynchronous runtime and tokio_mpmc::Queue to implement dual-queue relay), and the code is as follows:
[0035] The project's Cargo.toml file needs to include the following core dependencies:
[0036] [dependencies]
[0037] tokio = { version = "1.48.0", features = ["full"]} # Asynchronous runtime
[0038] tokio-mpmc = "0.2.4" # Multi-producer-multi-consumer queue
[0039] async-trait = "0.1.89" # Enables async methods in the trait
[0040] uuid = "1.18.1" # Generate a unique message ID
[0041] chrono = "0.4.42" # Timestamp processing
[0042] sqlx = { version = "0.8.6", features = ["mysql", "runtime-tokio-native-tls"]} # Database operations (for example)
[0043] Standardized interface layer definition (Rust trait):
[0044] Based on Rust's trait feature, three core interfaces are defined to standardize the behavior of the producer, consumer, and store components, ensuring that the components can be seamlessly integrated into the dual-queue system:
[0045] (1) Producer trait: contains abstract asynchronous methods for message generation (e.g., produce(&self) -> impl Future). <output = message>This stipulates that producers need to implement the message generation logic (such as reading data from the data source), without needing to worry about the message delivery method.
[0046] (2) Consumer trait: contains abstract asynchronous methods for message processing (e.g., consume(&self, msg: Message) -> impl Future). <Output = Result<ProcessedData, Error> The document stipulates that consumers must implement the business processing logic for messages (such as data cleaning and transformation), without needing to pay attention to the source of the messages.
[0047] (3) Storage trait: contains abstract asynchronous methods for data persistence (e.g., store(&self, data: ProcessedData) -> impl Future) <Output = Result<(), Error> The document stipulates that storage components must implement data persistence logic (such as writing to a database), without needing to concern themselves with the data source.
[0048] The above interfaces are all designed based on Rust's async / await asynchronous programming model, adapted to the scheduling mechanism of the tokio runtime, to ensure the efficient execution of asynchronous tasks. The code is as follows:
[0049] use async_trait::async_trait;
[0050] use std::future::Future;
[0051] / / Message structure (unified message format)
[0052] #[derive(Debug, Clone)]
[0053] pub struct Message {
[0054] pub id: String, / / Unique identifier for the message
[0055] pub payload: Vec <u8>, / / Message content (byte stream)
[0056] pub timestamp: u64, / / Generate timestamp (milliseconds)
[0057] }
[0058] / / Processed data structure (transfer medium for consumption → storage)
[0059] #[derive(Debug, Clone)]
[0060] pub struct ProcessedData {
[0061] pub msg_id: String, / / The associated message ID
[0062] pub content: String, / / The processed content
[0063] }
[0064] / / Producer interface: Defines message generation logic
[0065] #[async_trait]
[0066] pub trait Producer: Send + Sync + 'static {
[0067] async fn produce(&self) -> Message;
[0068] }
[0069] / / Consumer interface: Defines message processing logic (outputs ProcessedData after processing, without directly calling storage)
[0070] #[async_trait]
[0071] pub trait Consumer: Send + Sync + 'static {
[0072] async fn consume(&self, msg: Message) -> Result<ProcessedData,String> ;
[0073] }
[0074] / / Storage interface: Defines the data persistence logic (retrieving ProcessedData from the queue)
[0075] #[async_trait]
[0076] pub trait Storage: Send + Sync + 'static {
[0077] async fn store(&self, data: ProcessedData) -> Result<(), String>;
[0078] }
[0079] Dual-queue message relay layer implementation (based on tokio_mpmc::Queue):
[0080] This implementation uses Rust's `tokio_mpmc::Queue` (a multi-producer, multi-consumer asynchronous queue), which contains two independent queues. These queues handle the "producer-consumer" and "consumer-storage" tasks respectively, serving as the core hub connecting the standardized interface layer and the replaceable component layer.
[0081] (1) Production-consumption queue: responsible for receiving and temporarily storing messages output by the producer interface, distributing them to the consumer interface according to the FIFO strategy, shielding the direct interaction between the producer and the consumer, and balancing the production and consumption rates.
[0082] (2) Consumption-Storage Queue: Responsible for receiving the processing results (ProcessedData) output by the consumer interface and temporarily storing them, distributing them to the storage interface according to the FIFO strategy, shielding the direct interaction between the consumer and the storage component, and balancing the consumption and storage rates.
[0083] Both queues support a multi-producer-multi-consumer model, leveraging the lightweight asynchronous communication capabilities of tokio_mpmc to achieve efficient message delivery and concurrent safety control. Furthermore, queue capacity can be independently configured to flexibly adapt to the rate requirements of different stages. The code is as follows:
[0084] use tokio_mpmc::{Sender, Receiver, queue};
[0085] Initialize two queues: a producer-consumer queue and a consumer-store queue.
[0086] / / / Return value: (Producer-Consumer Queue Sender, Producer-Consumer Queue Receiver, Consumer-Storage Queue Sender, Consumer-Storage Queue Receiver)
[0087] pub fn init_double_queue(
[0088] prod_cons_capacity: usize, / / Producer-consumer queue capacity
[0089] cons_store_capacity: usize / / Consumption-store queue capacity
[0090] ) -> (Sender <message>, Receiver <message>, Sender <processeddata>,Receiver <processeddata>) {
[0091] let prod_cons_queue = queue(prod_cons_capacity);
[0092] let cons_store_queue = queue(cons_store_capacity);
[0093] (prod_cons_queue.0, prod_cons_queue.1, cons_store_queue.0, cons_store_queue.1)
[0094] }
[0095] Example of a replaceable component layer (implementing Rust trait)
[0096] It contains concrete components that implement the above standardized interface (Rust trait). All components interact indirectly through queues and have no direct dependencies.
[0097] (1) Producer components: such as "log file producer" and "sensor data producer", which are connected to the system by implementing the Producertrait, focus on message generation and deliver messages to the producer-consumer queue.
[0098] (2) Consumer components: such as "data cleaning consumer" and "data analysis consumer", which focus on message processing by implementing Consumertrait access system and delivering the processing results to the consumption-storage queue.
[0099] (3) Storage components: such as "database storage components" and "distributed cache storage components," which integrate with the system through Storagetrait, focusing on data persistence and retrieving data to be stored from the consumer-storage queue. Specific code is as follows:
[0100] (1) Producer components (such as "log file producer"):
[0101] pub struct LogFileProducer {
[0102] file_path: String, / / Log file path
[0103] }
[0104] impl LogFileProducer {
[0105] pub fn new(file_path: String) -> Self {
[0106] Self { file_path}
[0107] }
[0108] }
[0109] #[async_trait]
[0110] impl Producer for LogFileProducer {
[0111] async fn produce(&self) -> Message {
[0112] / / Asynchronously read log file contents
[0113] let payload = tokio::fs::read_to_string(&self.file_path)
[0114] .await
[0115] .unwrap_or_else(|e| format!("read error: {}", e))
[0116] .into_bytes();
[0117] / / Encapsulate as a Message and return
[0118] Message {
[0119] id: uuid::Uuid::new_v4().to_string(),
[0120] payload,
[0121] timestamp: chrono::Utc::now().timestamp_millis() as u64,
[0122] }
[0123] }
[0124] }
[0125] (2) Consumer components (e.g., "log scrubbing consumer"):
[0126] pub struct LogCleaner;
[0127] #[async_trait]
[0128] impl Consumer for LogCleaner {
[0129] async fn consume(&self, msg: Message) -> Result<ProcessedData,String> {
[0130] / / Message processing: Byte stream to string and cleanup
[0131] let content = String::from_utf8(msg.payload)
[0132] .map_err(|e| format!("invalid utf8: {}", e))?;
[0133] let cleaned_content = content.replace("\r\n", " ").trim().to_string();
[0134] Ok(ProcessedData {
[0135] msg_id: msg.id,
[0136] content: cleaned_content,
[0137] })
[0138] }
[0139] }
[0140] (3) Storage components (such as "MySQL Storage"):
[0141] pub struct MysqlStorage {
[0142] db_url: String, / / Database connection URL
[0143] }
[0144] impl MysqlStorage {
[0145] pub fn new(db_url: String) -> Self {
[0146] Self { db_url}
[0147] }
[0148] }
[0149] #[async_trait]
[0150] impl Storage for MysqlStorage {
[0151] async fn store(&self, data: ProcessedData) -> Result<(), String>{
[0152] / / Asynchronously connect to the database and insert data
[0153] let pool = sqlx::mysql::MySqlPool::connect(&self.db_url)
[0154] .await
[0155] .map_err(|e| format!("db connect error: {}", e))?;
[0156] sqlx::query!(
[0157] "INSERT INTO processed_logs (msg_id, content) VALUES (?,?)",
[0158] data.msg_id,
[0159] datacontent )
[0161] .execute(&pool)
[0162] .await
[0163] .map_err(|e| format!("db insert error: {}", e))?;
[0164] Ok(())
[0165] }
[0166] }
[0167] System Coordination Logic
[0168] During system initialization, based on Rust's type system and tokio task scheduling mechanism, the system coordinator (built-in logic) automatically completes the binding of components to the dual queues:
[0169] (1) Bind the producer component to the Sender of the producer-consumer queue to support multiple producers sending messages to the queue at the same time;
[0170] (2) Bind the consumer component to the Receiver of the production-consumption queue and the Sender of the consumption-storage queue. The consumer obtains messages from the production-consumption queue, processes them, and then delivers the results to the consumption-storage queue.
[0171] (3) Bind the storage component to the Receiver of the consumer-storage queue to support multiple storage components to obtain data from the queue and store it in parallel.
[0172] Users only need to develop components that conform to the trait specification, and they can automatically integrate into the system through the coordinator, achieving "plug and play" functionality. There's no need to worry about the details of asynchronous queue communication or asynchronous scheduling. The code is as follows:
[0173] use std::sync::Arc;
[0174] pub struct ArchitectureCoordinator {
[0175] producer: Arc<dyn Producer> / / Producer component
[0176] consumer: Arc<dyn Consumer> / / Consumer component
[0177] storage: Arc<dyn Storage> , / / Storage component
[0178] prod_cons_sender: Sender <message> / / Sender, the producer-consumer queue
[0179] prod_cons_receiver: Receiver <message> / / Receiver (producer-consumer queue)
[0180] cons_store_sender: Sender <processeddata> / / Consumer-Storage Queue Sender
[0181] cons_store_receiver: Receiver <processeddata> / / Receiver for consuming and storing data
[0182] }
[0183] impl ArchitectureCoordinator {
[0184] / / / Initialize the coordinator: bind components and double queues
[0185] pub fn new(
[0186] producer: impl Producer,
[0187] consumer: impl Consumer,
[0188] storage: impl Storage,
[0189] prod_cons_capacity: usize, / / Producer-consumer queue capacity
[0190] cons_store_capacity: usize / / Consumption-store queue capacity
[0191] ) -> Self {
[0192] let (prod_cons_sender, prod_cons_receiver, cons_store_sender,cons_store_receiver) =
[0193] init_double_queue(prod_cons_capacity, cons_store_capacity);
[0194] Self {
[0195] producer: Arc::new(producer),
[0196] consumer: Arc::new(consumer),
[0197] storage: Arc::new(storage),
[0198] prod_cons_sender,
[0199] prod_cons_receiver,
[0200] cons_store_sender,
[0201] cons_store_receiver,
[0202] }
[0203] }
[0204] / / / Startup System: End-to-End Asynchronous Flow (Producer → Production-Consumer Queue → Consumer → Consumption-Storage Queue → Storage)
[0205] pub async fn run(self) {
[0206] / / 1. Producer task: Generate messages and deliver them to the producer-consumer queue.
[0207] let producer = self.producer.clone();
[0208] let mut prod_cons_sender = self.prod_cons_sender.clone();
[0209] tokio::spawn(async move {
[0210] loop {
[0211] let msg = producer.produce().await;
[0212] prod_cons_sender.send(msg).await.expect("send toprod-cons queue failed");
[0213] }
[0214] });
[0215] / / 2. Consumer task: Retrieve messages from the producer-consumer queue → Process → Deliver to the consumer-storage queue
[0216] let consumer = self.consumer.clone();
[0217] let mut prod_cons_receiver = self.prod_cons_receiver;
[0218] let mut cons_store_sender = self.cons_store_sender.clone();
[0219] tokio::spawn(async move {
[0220] while let Some(msg) = prod_cons_receiver.recv().await {
[0221] match consumer.consume(msg).await { <00%00451>Ok(processed_data) => {
[0223] cons_store_sender.send(processed_data).await.expect("send to cons - store queue failed");
[0224] }
[0225] Err(e) => eprintln!("consumer process failed:{}", e),
[0226] }
[0227] [[ID=2%]]
[0228] });
[0229] / / 3. Storage task: Get data from the consumption - storage queue → Persist
[0230] let storage = self.storage.clone();
[0231] let mut cons_store_receiver = self.cons_store_receiver;
[0232] tokio::spawn(async move {
[0233] [[ID=4%]] while let Some(processed_data) = cons_store_receiver.recv().await {
[0234] if let Err(e) = storage.store(processed_data).await {
[0235] eprintln!("storage failed: {}", e);
[0236] / / Scalable: Deliver failed data to a dead-letter queue to support subsequent retries
[0237] }
[0238] }
[0239] });
[0240] }
[0241] }
[0242] Usage process (from the user's perspective)
[0243] When users develop new asynchronous systems based on this system, they only need to:
[0244] (1) Implement the Producer, Consumer, and Storage traits in Rust according to business requirements (focus on core business logic, without worrying about queue communication or asynchronous scheduling).
[0245] (2) Bind custom components and queues through ArchitectureCoordinator::new(...) and configure the capacity of the dual queues (adapt to the speed of each stage as needed);
[0246] (3) Calling run().await to start the system will automatically complete the entire asynchronous processing chain of "production → production-consumption queue → consumption → consumption-storage queue → storage". The code is as follows:
[0247] #[tokio::main]
[0248] async fn main() {
[0249] / / 1. Instantiate custom components (users only need to implement traits, without needing to worry about queues or scheduling)
[0250] let producer = LogFileProducer::new(" / var / log / app.log".to_string());
[0251] let consumer = LogCleaner;
[0252] let storage = MysqlStorage::new("mysql: / / user:pass@localhost:3306 / log_db".to_string());
[0253] / / 2. Initialize the coordinator (configure dual queue capacity)
[0254] let coordinator = ArchitectureCoordinator::new(
[0255] producer
[0256] consumer
[0257] storage,
[0258] 1000, / / Production-consumption queue capacity (adapted to production rate)
[0259] 500 / / Consumption-Storage Queue Capacity (Adapted to Storage Rate) );
[0261] / / 3. Start the system (automatic workflow across the entire process)
[0262] coordinator.run().await;
[0263] }
[0264] The prior description of the invention is provided to enable any person skilled in the art to make or use the invention. Various modifications to the invention will be apparent to those skilled in the art, and the general principles defined herein can be applied to other variations without departing from the spirit or scope of the invention. Thus, the invention is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0265] The embodiments of the present invention have been described in detail above. The description of the embodiments above is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.< / processeddata> < / processeddata> < / message> < / message> < / processeddata> < / processeddata> < / message> < / message> < / output>
Claims
1. A fully decoupled asynchronous processing system based on dual queues, characterized in that, It includes a standardized interface layer, a dual-queue message relay layer, and a replaceable component layer: The standardized interface layer defines multiple core interfaces based on Rust's trait features, standardizes the behavior of components, and ensures that the components can be seamlessly connected to the dual-queue message relay layer. The component refers to a module that implements a certain standard trait and can be replaced or combined to complete a certain responsibility. The dual-queue message relay layer is connected to the standardized interface layer. The dual-queue message relay layer is implemented based on Rust's tokio_mpmc::Queue (multi-producer-multi-consumer asynchronous queue) and contains two independent queues, each of which undertakes different relay tasks. The replaceable component layer is connected to the dual-queue message relay layer. The replaceable component layer contains the components that implement the standardized interface layer. All the components interact indirectly through the independent queue and have no direct dependencies. The three layers depend only on abstraction and queues and not on each other, thereby achieving full-link decoupling and allowing any of the components to be plug-and-play.
2. The end-to-end decoupled asynchronous processing system according to claim 1, characterized in that, The multiple core interfaces include a producer interface, which contains an abstract asynchronous method for message generation. The producer is required to implement the message generation logic but does not need to worry about the message transmission method.
3. The end-to-end decoupled asynchronous processing system according to claim 1, characterized in that, The multiple core interfaces also include a consumer interface, which contains an abstract asynchronous method for message processing. This method requires consumers to implement the business logic for message processing without needing to concern themselves with the source of the message.
4. The end-to-end decoupled asynchronous processing system according to claim 1, characterized in that, The multiple sets of core interfaces also include a storage interface, which contains an abstract asynchronous method for data persistence, stipulating that storage must implement data persistence logic without needing to concern itself with the data source.
5. The end-to-end decoupled asynchronous processing system according to claim 1, 2, or 3, characterized in that, The independent queue includes a producer-consumer queue, which is responsible for receiving and temporarily storing messages output by the producer interface, and distributing them to the consumer interface according to the FIFO (first-in, first-out) strategy, thus shielding the direct interaction between the producer and the consumer and balancing the production and consumption rates.
6. The end-to-end decoupled asynchronous processing system according to claim 1, 2, or 4, characterized in that, The independent queue also includes a consumer-storage queue, which is responsible for receiving and temporarily storing the processing results output by the consumer interface, distributing them to the storage interface according to the FIFO strategy, shielding the direct interaction between the consumer and the storage, and balancing the consumption and storage rates.
7. The end-to-end decoupled asynchronous processing system according to claim 5, characterized in that, The components include a producer component, which connects to the system by implementing the Producer trait. The producer component is used to generate messages and deliver the messages to the producer-consumer queue.
8. The end-to-end decoupled asynchronous processing system according to claim 6, characterized in that, The components also include a consumer component, which accesses the system by implementing the Consumer trait. The consumer component is used to process messages and deliver the processing results to the corresponding consumer-storage queue.
9. The end-to-end decoupled asynchronous processing system according to claim 6, characterized in that, The component also includes a storage component, which accesses the system by implementing the Storage trait, obtains data to be stored from the consumption-storage queue, and is used for data persistence.
10. The end-to-end decoupled asynchronous processing system according to claim 1, characterized in that, It also includes a system coordinator, which is responsible for binding the components to the independent queues and starting the corresponding asynchronous tasks to drive the automatic flow of messages throughout the entire chain. Users only need to develop components that conform to the trait specification, and they can automatically access the system through the coordinator without having to worry about the details of asynchronous communication or asynchronous scheduling of the queues.