System for efficient processing and queuing of data in a distributed microservices architecture with Azure Service Bus

Abstract

A system (100) for processing and queuing data in a distributed microservices architecture using Azure Service Bus, including: a) a message production module for generating and publishing metadata-enriched messages to Azure Service Bus topics; b) a message routing module for directing messages to appropriate subscriptions using metadata filters and rules; c) a message consumption module configured to consume and process messages from subscriptions, with support for session-based ordering; d) a retry and backoff module for retrying failed messages using exponential backoff and postponement strategies; e) a dead-letter handling module for checking, categorising and resolving dead-letter messages; f) a subscription management module for dynamically configuring topic-subscription relationships; g) a telemetry and monitoring module for collecting, analyzing and reporting message processing metrics; h) an access control module to manage secure communication and authorization between modules and the Azure service bus.

Description

The present invention relates to distributed computing and cloud-native. More particularly, it relates to systems for processing and queuing data across modular microservices. The invention uses Azure Service Bus for reliable, scalable and secure communication between the services.In modern cloud-native applications, microservices have become architectural standard due to their ability to isolate, scale independently, and speed up provisioning cycles. However, as the number of microservices increases, the complexity of managing communication therebetween also increases, particularly in asynchronous and event driven scenarios. Conventional messaging systems are often unable to address challenges such as message ordering, fault tolerance, reliable delivery, and dynamic service coordination. These constraints may result in loss of messages, processing bottleneck, increased latency, and difficulties in scaling, monitoring, and recovery from failures.A major problem arises when multiple microservices need to interact via queues and topics without tight coupling. Most systems lack a robust mechanism for message filtering, session tracking, reiteration handling, and message forwarding in a large scale. Moreover, the configuration and maintenance of consistent and secure message flows becomes a non-trivial task when services are scaled horizontally or introduced dynamically. Existing architectures often require significant manual intervention, have poor observability, or rely on duplicated user-defined logic across services to handle repetitions and errors.The present invention solves these problems by introducing a modular cloud-native system that utilizes Azure Service Bus to provide a reliable, serviceable, and scalable messaging backbone. Each module in the system is responsible for a particular function, such as generating, forwarding, consuming, repeating, or monitoring messages. By encapsulating these responsibilities, the system reduces complexity, improves fault tolerance, and ensures comprehensible, secure, and efficient message processing across all service interactions.An object of the present disclosure is to improve scalability and flexibility through a fully modular architecture.Another object of the present disclosure is to ensure reliable messaging using the enhanced Azure Service Bus functions.Another object of the present disclosure is to maintain message ordering and session consistency across distributed modules.Another object of the present disclosure is to enable automatic fault handling and recovery via dead letter and retry modules.Another object of the present disclosure is to support dynamic routing and prioritization based on message metadata.Another object of the present disclosure is to enable real-time monitoring and traceability with integrated telemetry.Another object of the present disclosure is to simplify service onboarding with dynamic subscription management.Another object of the present disclosure is to enhance security through role-based access control and encryption.Other objects and advantages of the present disclosure will become apparent from the following description, which is not intended to limit the scope of the present disclosure.The present invention relates to a system for efficiently processing and queuing data within a distributed microservices architecture using Azure Service Bus. It introduces a number of specialized modules that each handle a particular aspect of the messaging workflow, such as message production, routing, consumption, retry logic, dead letter handling, subscription management, telemetry, and access control. By utilizing the native functions of Azure Service Bus, such as topic-based routing, sessions, dead letter queues, and metadata filtering, the system ensures reliable, ordered, and secure messaging between loosely coupled services. The system allows dynamic scaling, independent module provisioning, and fault isolation, while simplifying observability and government.The present invention relates to a system for efficient data queuing and processing in an Azure Service Bus distributed microservices architecture. It comprises various modules such as message producer, message routing, message consumer, retry and backoff, dead-letter handling, subscription management, telemetry monitoring and access control. Each module performs a specific task to ensure reliable, ordered and secure communication. The system supports dynamic scaling, independent provisioning, and clear fault isolation. This increases system reliability, serviceability, and performance in complex cloud environments.The system consists of a collection of modules that can be used independently of one another and each take on a special role in the distributed messaging workflow.The message routing module acts as an intermediary that applies Azure Service Bus rules and filters to dynamically route incoming messages to the corresponding topic subscriptions. This module interprets message headers and uses SQL-like expressions to route messages to the proper destination modules based on logic such as service type, priority level, or processing context. The subscriptions are dynamically configured to enable real-time scalability of the consumer modules without interrupting the message flow.The message consumer module subscribes to one or more Azure Service Bus subscriptions and processes incoming messages based on configured workflows. This module uses session-activated subscriptions to maintain ordered message delivery when needed and ensure that related messages are processed in the proper order. The message consumer module is designed stateless and can be scaled horizontally to handle different workloads.To cope with message processing errors, the system includes a retry and backoff module incorporated into the Azure Service Bus delivery count function. This module implements exponential backoff strategies and techniques for moving messages to prevent message flooding and provide sufficient time for dependent modules to restore. If the retry thresholds are exceeded, the message is forwarded to a dead letter queue.The dead letter handling module continuously monitors the dead letter queues of Azure Service Bus and applies remedial logic to failed messages. It categorizes dead letter messages based on the fault type and decides whether to be reprocessed, discarded, or escalated based on business rules. This module ensures that no message is permanently lost without proper analysis and optional recovery.The subscription management module regulates the life cycle of topic subscriptions. Modules allow dynamic registration, deregistration and updating of their subscriptions depending on availability and workload. This facilitates modular scaling and decouples dependencies on services, thereby reducing the risk of bottleneck or unavailability of services.For observability, the telemetry and monitoring module collects detailed operational metrics such as message latency, processing time, failure rates, and queue depths. This module can be integrated with Azure Monitor, Log Analytics, and Application Insights to present dashboards in real-time and to send alerts based on adjustable thresholds. All modules issue structured telemetry protocols that allow extensive traceability from message creation to processing.Security is managed via the access control module passing through identity and access management policies using Azure Active Directory and Role-Based Access Control (RBAC). This module ensures that each module within the system has the appropriate permissions to interact with certain service bus entities and that all messages are encrypted during transmission and in the idle state.The invention is explained again below with reference to the figure. The following shows: FIG. 1 shows a modular system for efficient processing and classification of data into an Azure Service Bus distributed microservices architecture.In FIG. 1, a system ( 100) is shown. Operation of the system begins with the message producer module, which generates and publishes messages with structured metadata such as destination module identifiers, priority levels, and correlation IDs. These messages are forwarded to Azure Service Bus topics, where the message routing module evaluates each message using rule-based filters. Depending on the content of the message and the metadata assigned, the message is forwarded to one or more topic subscriptions. The subscription management module dynamically provides and updates these subscriptions to ensure that each destination module receives the proper messages based on current system configuration and service availability. This approach ensures decoupled and scalable communication between the modules.Once the messages have been forwarded, they are consumed by the message consumer module, which processes them according to the defined business logic. For ordered or related operations, session-activated queues are used to maintain consistency. If a message cannot be processed, the retry and retry module takes the task of applying exponential retry and retry. When the retries are exhausted, the message is forwarded to the dead letter queue, where the dead letter handling module analyzes the message and optionally processes it again. Throughout the process, the telemetry and supervisory module acquires real-time metrics while the access control module passes security and role-based access to all service bus units, thus ensuring secure and conformal message processing throughout the system.

Claims

A system (100) for processing and queuing data in a distributed microservices architecture using Azure Service Bus, including: a) a message production module for generating and publishing metadata enriched messages to Azure Service Bus topics; b) a message routing module for routing messages to appropriate subscriptions using metadata filters and rules; c) a message consumption module configured to consume and process messages from subscriptions, with assistance for session-based ordering; d) a retry and backoff module for retrying failed messages using exponential backoff and relocation strategies; e) a dead letter handling module for checking, categorizing and remedying dead letter messages; f) a subscription management module for dynamically configuring topic subscription relationships; g) a telemetry and monitoring module for collecting, analyzing and reporting message processing metrics; h) an access control module for managing secure communication and authorization between modules and the Azure service bus.The system (100) of claim 1, wherein the message routing module applies rules based on user-defined message headers including service identifier, correlation ID, and priority level.The system (100) of claim 1, wherein the message consumer module supports parallel processing through stateless design and horizontal scaling.The system (100) of claim 1, wherein the retry and backoff module limits the number of message repetitions using Azure Service Bus delivery and applies an exponential backoff pattern to reduce resource loading.The system (100) of claim 1, wherein the dead letter handling module automatically categorizes messages and applies remedial workflows based on fault type and business policy.The system (100) of claim 1, wherein the telemetry and monitoring module is integrated into azure's own logging tools and provides real-time dashboards to system administrators.The system (100) of claim 1, wherein the access control module passes message-level security through Azure Active Directory and end-to-end encryption policies.

Images