Message-oriented middleware high-availability data transmission method and system

Abstract

The invention discloses a message middleware high-availability data transmission method and system. The method comprises the steps that a message producer initiates a write-in request to a file manager; the file manager inquires an internal metadata index table and returns an effective write-in address of the file slot to the message middleware client library; the message middleware client library directly writes the message data stream into a specified file slot on the distributed file storage layer; the file manager updates an internal metadata index table and returns a confirmation of writing success to the message middleware client library; and the message middleware client library returns the confirmation of writing success to the message producer. According to the scheme, triple backup of the local file, the middleware and the database and file landing compensation are achieved, the resource occupancy peak value is reduced by 40% through dynamic scheduling, and the problems that in the prior art, files are prone to being lost, compensation tasks and service resources compete with each other, and a transmission data integrity verification mechanism is lacked are solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a high-availability data transmission method and system for message middleware. Background Technology

[0002] In distributed systems, message brokers (such as Kafka and RocketMQ) serve the functions of decoupling and asynchronous communication, but their high availability is limited by network and hardware stability. Message brokers are software systems that provide synchronous, asynchronous, and reliable message transmission support for application systems in a network environment, based on queue and message passing technologies. [1] Its primary function is to facilitate data exchange between cross-platform and cross-protocol application systems in a distributed environment. It extends inter-process communication through message queues, ensuring reliable message storage and forwarding. Existing disaster recovery solutions often employ primary / standby failover or retry queues, but these do not address the issue of data persistence when the middleware is completely unavailable.

[0003] Existing solutions primarily address data transmission interruptions caused by single points of failure in message middleware clusters. They typically include the following components:

[0004] Message producer cluster: Business systems send messages to middleware;

[0005] Master-standby message middleware nodes: Deployed in Active-Standby mode (such as RocketMQ master-slave architecture);

[0006] Status monitoring module: Real-time detection of node health status (heartbeat detection cycle: 5s);

[0007] Master / standby switchover module: elects a new master node based on ZooKeeper.

[0008] The data transmission process includes:

[0009] The producer sends messages to the in-memory queue of the current master node (Active Node);

[0010] The master node asynchronously replicates messages to the standby node.

[0011] If the master node response times out (threshold: 3s), the status monitoring module will trigger an alarm.

[0012] The primary / standby switchover module promotes the standby node to the new primary node (switchover time: 8-10 seconds).

[0013] The producer reconnects to the new master node and resends the unacknowledged messages in memory.

[0014] Data persistence mechanisms include:

[0015] Messages are stored only in memory queues and the middleware’s built-in storage (such as Kafka’s partition logs).

[0016] The lack of a local file layer design results in message loss due to non-persistent message handling during failures.

[0017] In existing technologies, data is stored only in content or databases, making it easy to lose in case of failure; compensation tasks compete with business resources, and there is no dynamic compensation mechanism; at the same time, there is a lack of a mechanism to verify the integrity of transmitted data. Summary of the Invention

[0018] The purpose of this invention is to provide a high-availability data transmission method and system for message middleware, which realizes triple backup and end-to-end compensation of local files, middleware, and database, adds a local file persistence layer, and dynamically schedules compensation timing to solve the problems of easy file loss, competition between compensation tasks and business resources, and lack of data integrity verification mechanism in existing technologies.

[0019] To achieve the above objectives, embodiments of this application provide a high-availability data transmission method for message middleware, comprising:

[0020] The message producer sends a write request to the file manager through its built-in message middleware client library;

[0021] After receiving the request, the file manager queries its internal metadata index table to find the file slot currently being written to in the corresponding partition and its node address; it then returns the valid write address of the file slot to the message middleware client library.

[0022] After obtaining the write address, the message middleware client library bypasses the file manager and directly writes the message data stream to the specified file slot on the distributed file storage layer;

[0023] Once the write operation is successfully completed, the message middleware client library notifies the file manager again;

[0024] The file manager updates its internal metadata index table and returns a confirmation of successful write to the message middleware client library;

[0025] The message middleware client library returns a confirmation of successful write to the message producer, completing the entire write process.

[0026] When the file manager's status probe detects that a file slot is unresponsive or has timed out during a scheduled health check, the following steps are performed:

[0027] The file manager marks faulty file slots in the metadata index table as unavailable file slots;

[0028] The file manager's embedded partition controller will allocate a new file slot for the corresponding partition and update the address of the new file slot in the metadata index table;

[0029] The file manager returns the new write address to subsequent write requests; and marks unavailable file slots as file slots to be compensated.

[0030] The file persistence compensation device built into the file manager periodically scans the metadata index table. When it finds a file slot to be compensated, it directly accesses the distributed file storage layer, checks the last state of the file slot, and performs a repair operation to ensure that no data is corrupted or unclosed. After the compensation is completed, it notifies the file manager.

[0031] The file manager updates the status of the file slot in the metadata index table to either a closed file slot or an archived file slot; the closed or archived file slot can then be read normally as historical data.

[0032] The file persistence compensation device is connected in series between the message producer and the message middleware to achieve bidirectional data interception. It is a lightweight background process or thread embedded in the file manager.

[0033] By periodically scanning the metadata index table, file slots that are being written to but have not been updated for a long time are identified, indicating that their original write nodes may have failed; the finalization operation of the corresponding file slot is completed directly on the distributed file storage layer, and the file manager is notified to update the metadata index table.

[0034] The compensation scheduler is built into the file-to-deployment compensation device and is responsible for fault handling. It monitors the message middleware cluster in real time through status probes. When an anomaly is detected in the message middleware, the compensation scheduler reads the unsent messages from the distributed file storage layer and resends them to the message middleware. The monitoring period is 1 second, and the preferred collection items are CPU utilization, IO wait, and network latency.

[0035] The compensation weight value is calculated based on system load and message priority; the retry priority index W = (100 - CPU utilization) × priority coefficient is dynamically calculated; where the high priority coefficient is preferably 2.0 and the low priority coefficient is preferably 0.5.

[0036] When a consumer specifies the partition to consume and the starting offset through its built-in message middleware client library, the message middleware client library queries the file manager for the physical file location where the offset message is located;

[0037] The file manager receives the request, queries the metadata index table, locates the file slot corresponding to the message based on the offset, obtains the complete access path of the file slot on the distributed file storage layer, and returns the complete path to the message middleware client library.

[0038] After obtaining the file path, the message middleware client library directly reads the file data stream from the specified location on the distributed file storage layer, parses out the specific message content, and returns the parsed message to the consumer.

[0039] The file manager is responsible for managing all message metadata, allocating write locations, load balancing, and fault handling, and contains several key components:

[0040] Metadata index table: Records the latest file path, offset, status, and other information for each message partition or topic;

[0041] Partition Controller: Built into the file manager. It is responsible for allocating and managing specific file write node file slots for each message partition.

[0042] Status probe: Built into the file manager. It proactively and periodically initiates health checks on each file slot to monitor its availability and load status;

[0043] The file manager is responsible for storing messages. It distributes data to 128 file slots (the number of file slots is not a static value and needs to be iteratively optimized based on operation monitoring data) in the distributed file storage layer according to the business ID hash value through the partition controller. The file slot has a maximum file size of 10GB. The file manager also updates the metadata index table to record the message status.

[0044] The message middleware client library is embedded in the message producer and message consumer applications, encapsulates the communication protocol with the system core, and provides simple message sending and fetching interfaces for the applications; it also embeds a partition controller and is responsible for communicating with the remote file manager.

[0045] A file slot is not an independent module, but a logical concept on the distributed file storage layer. It is a file segment that is allocated to a specific message partition on the distributed file storage layer and is currently being written to. As the final destination of a message on the disk, it is the smallest unit for performing a physical write operation.

[0046] Once a file slot is full, it becomes a read-only archive file, which can be read by consumers or cleaned up by the system according to the retention policy.

[0047] On the other hand, this application provides a message middleware high-availability data transmission system, including: a message producer, a client located outside the system, a message sender and a receiver, which interact with the system core through an introduced message middleware client library;

[0048] The message middleware client library is embedded in message producer and consumer applications. It encapsulates the communication protocol with the system core and provides simple message sending and fetching interfaces for applications. It also has an embedded partition controller and is responsible for communicating with the remote file manager.

[0049] The file manager is a core server-side component of the system, deployed independently within a service cluster. It manages all message metadata, allocates write locations, performs load balancing, and handles faults. Internally, it contains several key components:

[0050] Metadata index table: Records the latest file path, offset, status and other information for each message partition or topic, and is maintained and updated by the file manager itself;

[0051] Partition Controller: Built into the file manager, it is responsible for allocating and managing specific file write node file slots for each message partition;

[0052] Status probe: Built into the file manager, it proactively and periodically initiates health checks on each file slot to monitor its availability and load status;

[0053] A file slot is not an independent module, but a logical concept on the distributed file storage layer. It is a file segment that is allocated to a specific message partition on the distributed file storage layer and is currently being written. As the final destination of a message on the disk, it is the smallest unit for performing a physical write operation. When a file slot is full, it becomes a read-only archive file, which can be read by consumers or cleaned up by the system according to the recovery retention strategy.

[0054] The distributed file storage layer is the underlying infrastructure of the system, providing the final file persistent storage capability; the file manager maps file slots to the specific physical path of the distributed file storage layer; the file persistence compensation device performs file operations directly on this layer;

[0055] The file persistence compensation device is a lightweight background process or thread embedded in the file manager, rather than a standalone service. It periodically scans the metadata index table to identify file slots that are being written to but have not been updated for a long time, determining that their original write nodes may have failed. It then performs the finalization operation of the file slot directly on the distributed file storage layer and notifies the file manager to update the metadata index table to ensure data consistency.

[0056] The message producer sends a write request to the file manager through its built-in message middleware client library;

[0057] After receiving the request, the file manager queries its internal metadata index table to find the file slot currently being written to in the corresponding partition and its node address; it then returns the valid write address of the file slot to the message middleware client library.

[0058] After obtaining the write address, the message middleware client library bypasses the file manager and directly writes the message data stream to the specified file slot on the distributed file storage layer;

[0059] Once the write operation is successfully completed, the message middleware client library notifies the file manager again;

[0060] The file manager updates its internal metadata index table and returns a confirmation of successful write to the message middleware client library;

[0061] The message middleware client library returns a confirmation of successful write to the message producer, completing the entire write process.

[0062] When the file manager's status probe detects that a file slot is unresponsive or has timed out during a scheduled health check, the following steps are performed:

[0063] The file manager marks faulty file slots in the metadata index table as unavailable file slots;

[0064] The file manager's embedded partition controller will allocate a new file slot for the corresponding partition and update the address of the new file slot in the metadata index table;

[0065] The file manager returns the new write address to subsequent write requests; and marks unavailable file slots as file slots to be compensated.

[0066] The file persistence compensation device built into the file manager periodically scans the metadata index table. When it finds a file slot to be compensated, it directly accesses the distributed file storage layer, checks the last state of the file slot, and performs a repair operation to ensure that no data is corrupted or unclosed. After the compensation is completed, it notifies the file manager.

[0067] The file manager updates the status of the file slot in the metadata index table to either a closed file slot or an archived file slot; the closed or archived file slot can then be read normally as historical data.

[0068] When a consumer specifies the partition to consume and the starting offset through its built-in message middleware client library, the message middleware client library queries the file manager for the physical file location where the offset message is located;

[0069] The file manager receives the request, queries the metadata index table, locates the file slot corresponding to the message based on the offset, obtains the complete access path of the file slot on the distributed file storage layer, and returns the complete path to the message middleware client library.

[0070] After obtaining the file path, the message middleware client library directly reads the file data stream from the specified location on the distributed file storage layer, parses out the specific message content, and returns the parsed message to the consumer.

[0071] The method and system provided in this application include a file persistence compensation device connected in series between the producer and middleware to achieve bidirectional data interception; a distributed file storage layer deployed independently, communicating with the compensation device via TCP protocol; and a control link transmitting heartbeat detection signals (at 1-second intervals) and compensation trigger commands. This achieves triple backup of local files, middleware, and database, as well as file persistence compensation, and direct reading and retransmission of local files. Compared with existing technologies, dynamic scheduling reduces peak resource consumption by 40%, solving the problems of easy file loss, competition between compensation tasks and business resources, and lack of data integrity verification mechanisms in existing technologies. Attached Figure Description

[0072] Figure 1 This is a flowchart of a message middleware high-availability data transmission method provided according to an embodiment of this application;

[0073] Figure 2 This is a schematic diagram of the structure of a message middleware high-availability data transmission system provided according to an embodiment of this application;

[0074] Figure 3 This is a schematic diagram of the structure of a message middleware high-availability data transmission system according to a specific embodiment of this application;

[0075] Figure 4 This is a schematic diagram of the system core structure and message flow of a message middleware high-availability data transmission scheme according to a specific embodiment of this application. Detailed Implementation

[0076] To better understand the present invention, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of the present invention and are not intended to limit the embodiments of the present invention. Various modifications can be made to the embodiments as long as the effects of the present invention are achieved.

[0077] Figure 1 The flowchart of a high-availability data transmission method for message middleware provided in an embodiment of this application is illustrated, wherein,

[0078] Step 101: The message producer sends a write request to the file manager through its built-in message middleware client library.

[0079] Step 102: After receiving the request, the file manager queries the internal metadata index table to find the file slot currently being written to in the corresponding partition and its node address; and returns the valid write address of the file slot to the message middleware client library.

[0080] Step 103: After obtaining the write address, the message middleware client library bypasses the file manager and directly writes the message data stream to the specified file slot on the distributed file storage layer.

[0081] Step 104: After the write operation is successfully completed, the message middleware client library notifies the file manager again.

[0082] Step 105: The file manager updates its internal metadata index table and returns a confirmation of successful write to the message middleware client library.

[0083] Step 106: The message middleware client library returns a confirmation of successful writing to the message producer, completing the entire writing process.

[0084] Specifically, if the file manager's status probe detects that a file slot is unresponsive or has timed out during a scheduled health check, the following steps are executed:

[0085] The file manager marks faulty file slots in the metadata index table as unavailable file slots;

[0086] The file manager's embedded partition controller will allocate a new file slot for the corresponding partition and update the address of the new file slot in the metadata index table;

[0087] The file manager returns the new write address to subsequent write requests; and marks unavailable file slots as file slots to be compensated.

[0088] The file persistence compensation device built into the file manager periodically scans the metadata index table. When it finds a file slot to be compensated, it directly accesses the distributed file storage layer, checks the last state of the file slot, and performs a repair operation to ensure that no data is corrupted or unclosed. After the compensation is completed, it notifies the file manager.

[0089] The file manager updates the status of the file slot in the metadata index table to either a closed file slot or an archived file slot; the closed or archived file slot can then be read normally as historical data.

[0090] The file persistence compensation device is connected between the message producer and the message middleware to achieve two-way data interception. It is a lightweight background process or thread embedded in the file manager.

[0091] By periodically scanning the metadata index table, file slots that are being written to but have not been updated for a long time are identified, indicating that their original write nodes may have failed; the finalization operation of the corresponding file slot is completed directly on the distributed file storage layer, and the file manager is notified to update the metadata index table.

[0092] The compensation scheduler is built into the file-to-deployment compensation device and is responsible for fault handling. It monitors the message middleware cluster in real time through status probes. When an anomaly is detected in the message middleware, the compensation scheduler reads the unsent messages from the distributed file storage layer and resends them to the message middleware. The monitoring period is 1 second, and the preferred collection items are CPU utilization, IO wait, and network latency.

[0093] The compensation weight value is calculated based on system load and message priority; the retry priority index W = (100 - CPU utilization) × priority coefficient is dynamically calculated; where the high priority coefficient is preferably 2.0 and the low priority coefficient is preferably 0.5.

[0094] When a consumer specifies the partition to consume and the starting offset through its built-in message middleware client library, the message middleware client library queries the file manager for the physical file location where the offset message is located;

[0095] The file manager receives the request, queries the metadata index table, locates the file slot corresponding to the message based on the offset, obtains the complete access path of the file slot on the distributed file storage layer, and returns the complete path to the message middleware client library.

[0096] After obtaining the file path, the message middleware client library directly reads the file data stream from the specified location on the distributed file storage layer, parses out the specific message content, and returns the parsed message to the consumer.

[0097] The file manager is responsible for managing all message metadata, allocating write locations, load balancing, and fault handling, and contains several key components:

[0098] Metadata index table: records the latest file path, offset, status, and other information for each message partition or topic;

[0099] Partition Controller: Built into the file manager. It is responsible for allocating and managing specific file write node file slots for each message partition.

[0100] Status probe: Built into the file manager. It proactively and periodically initiates health checks on each file slot to monitor its availability and load status;

[0101] The file manager is responsible for storing messages and distributes data to 128 file slots in the distributed file storage layer according to the business ID hash value through the partition controller. Each file slot has a file limit of 10GB. The file manager also updates the metadata index table to record the message status.

[0102] The message middleware client library is embedded in the message producer and message consumer applications, encapsulates the communication protocol with the system core, and provides simple message sending and fetching interfaces for the applications; it also embeds a partition controller and is responsible for communicating with the remote file manager.

[0103] A file slot is not an independent module, but a logical concept on the distributed file storage layer. It is a file segment that is allocated to a specific message partition on the distributed file storage layer and is currently being written to. As the final destination of a message on the disk, it is the smallest unit for performing a physical write operation.

[0104] Once a file slot is full, it becomes a read-only archive file, which can be read by consumers or cleaned up by the system according to the retention policy.

[0105] The entire solution is based on a complete message middleware high-availability data transmission system, such as... Figure 2 As shown, it includes:

[0106] Message producer 21, located on an external client, is the sender and receiver of messages, and interacts with the system core through the introduced message middleware client library 22;

[0107] The message middleware client library 22 is embedded in the message producer and consumer applications. It encapsulates the communication protocol with the system core and provides simple message sending and fetching interfaces for the application. It also has an embedded partition controller and is responsible for communicating with the remote file manager 23.

[0108] File Manager 23 is the core server-side component of the system, an independently deployed service cluster; it is responsible for managing all message metadata, allocating write locations, load balancing, and fault handling; internally, it contains several key devices:

[0109] Metadata index table: Records the latest file path, offset, status and other information for each message partition or topic, and is maintained and updated by the file manager itself;

[0110] Partition Controller: Built into the file manager, it is responsible for allocating and managing specific file write node file slots for each message partition;

[0111] Status probe: Built into the file manager, it proactively and periodically initiates health checks on each file slot to monitor its availability and load status;

[0112] File slot 24 is not an independent module, but a logical concept on the distributed file storage layer 25. It is a file fragment that is being written to and allocated to a specific message partition on the distributed file storage layer 25. As the final destination of the message on the disk, it is the smallest unit for performing physical write operations. When a file slot 24 is full, it becomes a read-only archive file, which can be read by consumers or cleaned up by the system according to the recovery retention strategy.

[0113] The distributed file storage layer 25 is the underlying infrastructure of the system, providing the final file persistent storage capability; the file manager 23 maps the file slots 24 to the specific physical paths of the distributed file storage layer 25; the file persistence compensation device 26 performs file operations directly on this layer.

[0114] The file persistence compensation device 26 is a lightweight background process or thread embedded in the file manager 23, rather than an independent service. By periodically scanning the metadata index table, it discovers file slots 24 that are being written to but have not been updated for a long time, and determines that their original write nodes may have failed. It directly completes the finalization operation of the file slot 24 on the distributed file storage layer 25 and notifies the file manager 23 to update the metadata index table to ensure data consistency.

[0115] Specifically, the file landing compensation device is connected in series between the producer and the middleware to achieve bidirectional data interception; the distributed file storage layer is deployed independently and communicates with the compensation device via TCP protocol; the control link transmits heartbeat detection signals (at 1-second intervals) and compensation trigger commands.

[0116] In one embodiment of the present invention, such as Figure 3 As shown, the file persistence compensation device is located between the message producer and the message middleware cluster, enabling direct write operations to the distributed file storage. Specifically,

[0117] 1. Producer / Consumer:

[0118] Location: External client of the system.

[0119] Function: The sender and receiver of messages. They interact with the system core through an introduced message middleware client library.

[0120] 2. Messaging Client Library:

[0121] Location: Embedded in producer and consumer applications.

[0122] Function: It encapsulates the communication protocol with the system kernel, providing applications with simple message sending and pulling interfaces. It embeds a partition controller and is responsible for communicating with the remote file manager.

[0123] 3. File Manager:

[0124] Location: The core server-side component of the system, deployed independently in a service cluster.

[0125] Function: The brain of the system, responsible for managing all message metadata, allocating write locations, load balancing, and fault handling. It contains several key components:

[0126] Metadata Index: Records the latest file path, offset, status, and other information for each message partition (or topic). This table is maintained and updated by the file manager itself.

[0127] Partition Controller: This device is built into the file manager. It is responsible for allocating and managing specific file write nodes (i.e., file slots) for each message partition.

[0128] Status Probe: This device is built into the file manager. It proactively and periodically initiates health checks on each file slot to monitor its availability and load status.

[0129] 4. File Slot:

[0130] Location: A file slot is not an independent module, but a logical concept on the distributed file storage layer. It is a file segment (such as a 1GB file) that is allocated to a specific message partition on the distributed file storage layer and is currently being written to.

[0131] Function: As the final destination of messages on disk, it is the smallest unit for performing physical write operations. Its subsequent function is: when a file slot is full, it becomes a read-only archive file, which can be read by consumers or cleaned up by the system according to the retention strategy.

[0132] 5. Distributed File Storage Layer:

[0133] Location: The underlying infrastructure of the system (such as HDFS, Ceph, or custom distributed object storage).

[0134] Function: Provides ultimate file persistent storage capability. The file manager maps file slots to specific physical paths on this storage layer. The file persistence compensation device performs file operations directly on this layer.

[0135] 6. File Write Compensation Agent:

[0136] Location: This is a lightweight background process or thread embedded in the file manager, rather than a standalone service.

[0137] Function: Acts as a safety mechanism. By periodically scanning the metadata index table, it identifies file slots marked as "writing" but not updated for an extended period, indicating that their original write nodes may have failed. Subsequently, the device directly performs the finalization operation for that file slot (such as closing the file handle, updating the file size, etc.) on the distributed file storage layer and notifies the file manager to update the metadata index table, ensuring data consistency.

[0138] The core operation process in the above embodiments includes the following aspects:

[0139] Process 1: Message Writing Process

[0140] 1. Step One (Initiating a Write Request):

[0141] Main body: Producers use their built-in message middleware client library.

[0142] Operation: The producer calls the `send` method of the client library. The partition controller in the client library calculates the target partition based on the message key and sends a write request to the file manager, which includes the target partition ID.

[0143] 2. Step Two (Allocating Write Resources):

[0144] Main component: File manager.

[0145] Operation: After receiving a request, the file manager queries its internal metadata index table to find the file slot currently being written to for that partition and its corresponding node address. Simultaneously, a status probe ensures that the file slot is in a healthy state.

[0146] Output: The file manager returns the valid write address (an accessible URL or path) of the file slot to the message middleware client library.

[0147] 3. Step Three (Perform Physical Write):

[0148] Main body: Message middleware client library (representing the producer).

[0149] Operation: After obtaining the write address, the client library bypasses the file manager and directly writes the message data stream (in batches) to the specified file slot on the distributed file storage layer.

[0150] 4. Step Four (Confirmation and Metadata Update):

[0151] Main components: message middleware client library and file manager.

[0152] Operation: Once the write operation is successfully completed, the client library will notify the file manager again. The file manager will then update its internal metadata index table, for example, by increasing the end-of-flight offset (EOF Offset) of the partition.

[0153] Output: The file manager returns an ACK to the client library confirming the successful write, and the client library then returns it to the producer, completing the entire write process.

[0154] Process Two: Fault Handling and Compensation Process

[0155] 1. Step One (Fault Discovery):

[0156] Scenario A: The file manager's status probe detects that a file slot is unresponsive or has timed out during a scheduled health check.

[0157] Scenario B: The message middleware client library fails in step three (performing a physical write), and it reports the failure to the file manager.

[0158] 2. Step Two (Isolation and Reassignment):

[0159] Main component: File manager.

[0160] Procedure: The file manager marks the faulty file slot in the metadata index table as "unavailable". Then, the partition controller allocates a new, healthy file slot (located on another storage node) for the partition and updates the address of the new file slot in the metadata index table.

[0161] 3. Step Three (Metadata Synchronization):

[0162] Main component: File manager.

[0163] Operation: The file manager returns the new write address to subsequent write requests (i.e., step two of process one). For older, faulty file slots, their status changes to "pending compensation".

[0164] 4. Step Four (Execute Compensation):

[0165] Main body: File landing compensation device (built into the file manager).

[0166] Operation: The device periodically scans the metadata index table and identifies file slots with a status of "pending compensation". It directly accesses the distributed file storage layer, checks the last state of the file slot, and performs necessary repair operations (such as truncating the file to the last valid offset and officially closing the file) to ensure that no data is corrupted or in an unclosed state.

[0167] Output: After compensation is complete, the file-to-file compensation device notifies the file manager, which then updates the status of the file slot in the metadata index table to "closed" or "archived." Afterward, the file slot can be read normally as historical data.

[0168] Step 3: Message Consumption Process

[0169] Step 1 (Initiating a pull request):

[0170] Main body: Consumers access the client through its built-in message middleware client library.

[0171] Operation: The consumer specifies the partition to consume and the starting offset. The client library queries the file manager for the physical file location of the offset message.

[0172] 2. Step Two (Querying Metadata):

[0173] Main component: File manager.

[0174] Operation: The file manager receives the request, queries the metadata index table, locates the file slot where the message is stored based on the offset (which may be the currently active slot being written to or the archived historical slot), and obtains the complete access path of that file slot on the distributed file storage layer.

[0175] Output: The file manager returns this path to the client's message middleware client library.

[0176] 3. Step Three (Perform Physical Read):

[0177] Main component: Message middleware client library (representing the consumer).

[0178] Operation: After obtaining the file path, the client library directly reads the file data stream from the specified location on the distributed file storage layer and parses out the specific message content from it.

[0179] Output: The client library returns the parsed message to the consumer.

[0180] like Figure 4 The diagram illustrates the core components and data flow of the file persistence compensation device. The message producer cluster sends messages to the file persistence compensation device. This device internally includes:

[0181] The file manager is responsible for storing messages. It distributes data to 128 file partitions (slots) in the distributed file storage layer based on the business ID hash value through the partition controller. Each partition has a file size limit of 10GB. The file manager also updates the metadata index table to record message status.

[0182] Compensation Scheduler: Responsible for fault handling, it monitors the message middleware cluster in real time via status probes (heartbeat interval of 1 second). When a middleware anomaly is detected, the compensation scheduler reads unsent messages from the distributed file storage layer (file slot paths are provided by the metadata index table) and resends them to the middleware.

[0183] Data flow: Producer → File persistence compensation device → File manager (storage) → Compensation scheduler (monitoring / retransmission) → Message middleware cluster. The control link is used to transmit heartbeat signals and compensation instructions.

[0184] In this embodiment, the file slot partitioning storage technology can store files by hash partitioning according to business ID, avoiding excessively large single files (e.g., each partition ≤ 10GB); the dynamic weight compensation algorithm can calculate compensation weight values ​​based on system load (CPU / IO) and message priority; the file-message joint verification mechanism can synchronously generate a metadata index table (including CRC32 and message status bits) during persistence; and the distributed file synchronization protocol can use the Raft consensus algorithm to synchronize file states across nodes. Compared with existing technical solutions, it has several technical advantages, as detailed in Table 1.

[0185] Table 1 Advantages compared to existing technologies

[0186]

[0187] Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or implemented by hardware or firmware through any other reasonable means of integrating or packaging circuits, or implemented by any one or a suitable combination of software, hardware, and firmware. Alternatively, one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0188] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0189] This invention provides a storage medium storing a program that, when executed by a processor, implements a message middleware high-availability data transmission method.

[0190] This invention provides a processor for running a program, wherein the program executes the message middleware high-availability data transmission method during runtime.

[0191] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0192] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0193] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0194] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0195] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0196] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0197] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0198] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0200] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0201] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0202] The accompanying drawings illustrate several block diagrams and / or flowcharts. It should be understood that some blocks, or combinations thereof, in the block diagrams and / or flowcharts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when executed by the processor, these instructions can create means for implementing the functions / operations described in these block diagrams and / or flowcharts. The technology of this application can be implemented in hardware and / or software (including firmware, microcode, etc.). Alternatively, the technology of this application can take the form of a computer program product stored on a computer-readable storage medium, which can be used by or in conjunction with an instruction execution system.

Claims

1. A high-availability data transmission method for message middleware, characterized in that, include: The message producer sends a write request to the file manager through its built-in message middleware client library; After receiving the request, the file manager queries the internal metadata index table to find the file slot currently being written to in the corresponding partition and its node address. Return the valid write address of the file slot to the message middleware client library; After obtaining the write address, the message middleware client library bypasses the file manager and directly writes the message data stream to the specified file slot on the distributed file storage layer; Once the write operation is successfully completed, the message middleware client library notifies the file manager again; The file manager updates its internal metadata index table and returns a confirmation of successful write to the message middleware client library; The message middleware client library returns a confirmation of successful write to the message producer, completing the entire write process.

2. The method according to claim 1, characterized in that, Also includes: When the file manager's status probe detects that a file slot is unresponsive or has timed out during a scheduled health check, the following steps are performed: The file manager marks faulty file slots in the metadata index table as unavailable file slots; The file manager's embedded partition controller will allocate a new file slot for the corresponding partition and update the address of the new file slot in the metadata index table; The file manager returns the new write address to subsequent write requests; Mark unavailable file slots as file slots to be compensated; The file persistence compensation device built into the file manager periodically scans the metadata index table. When it finds a file slot to be compensated, it directly accesses the distributed file storage layer, checks the last state of the file slot, and performs a repair operation to ensure that no data is in a corrupted or unclosed state. Notify the file manager after the compensation is complete; The file manager updates the status of the file slot in the metadata index table to either a closed file slot or an archived file slot; the closed or archived file slot can then be read normally as historical data.

3. The method according to claim 2, characterized in that, The file persistence compensation device is connected in series between the message producer and the message middleware to achieve bidirectional data interception. It is a lightweight background process or thread embedded in the file manager. By periodically scanning the metadata index table, file slots that are being written to but have not been updated for a long time are identified, indicating that their original write nodes may have failed; the finalization operation of the corresponding file slot is completed directly on the distributed file storage layer, and the file manager is notified to update the metadata index table. The compensation scheduler is built into the file-to-deployment compensation device and is responsible for fault handling. It monitors the message middleware cluster in real time through status probes. When an anomaly is detected in the message middleware, the compensation scheduler reads the unsent messages from the distributed file storage layer and resends them to the message middleware. The monitoring period is 1 second, and the preferred collection items are CPU utilization, IO wait, and network latency. The compensation weight value is calculated based on system load and message priority; the retry priority index W = (100 - CPU utilization) × priority coefficient is dynamically calculated; where the high priority coefficient is preferably 2.0 and the low priority coefficient is preferably 0.

5.

4. The method according to claim 1, characterized in that, The method further includes: When a consumer specifies the partition to consume and the starting offset through its built-in message middleware client library, the message middleware client library queries the file manager for the physical file location where the offset message is located; The file manager receives the request, queries the metadata index table, locates the file slot corresponding to the message based on the offset, obtains the complete access path of the file slot on the distributed file storage layer, and returns the complete path to the message middleware client library. After obtaining the file path, the message middleware client library directly reads the file data stream from the specified location on the distributed file storage layer, parses out the specific message content, and returns the parsed message to the consumer.

5. The method according to claim 1, characterized in that, The file manager is responsible for managing all message metadata, allocating write locations, load balancing, and fault handling, and contains several key components: Metadata index table: records the latest file path, offset, status, and other information for each message partition or topic; Partition Controller: Built into the file manager. It is responsible for allocating and managing specific file write node file slots for each message partition. Status probe: Built into the file manager. It proactively and periodically initiates health checks on each file slot to monitor its availability and load status; The file manager is responsible for storing messages and distributes data to file slots in the distributed file storage layer according to the business ID hash value through the partition controller. The file slot has a maximum file size of 10GB. The file manager also updates the metadata index table and records the message status.

6. The method according to claim 1, characterized in that, The message middleware client library is embedded in the message producer and message consumer applications, encapsulates the communication protocol with the system core, and provides simple message sending and fetching interfaces for the applications; it also embeds a partition controller and is responsible for communicating with the remote file manager. A file slot is not an independent module, but a logical concept on the distributed file storage layer. It is a file segment that is allocated to a specific message partition on the distributed file storage layer and is currently being written to. As the final destination of a message on the disk, it is the smallest unit for performing a physical write operation. Once a file slot is full, it becomes a read-only archive file, which can be read by consumers or cleaned up by the system according to the retention policy.

7. A high-availability data transmission system for message middleware, characterized in that, include: The message producer, located on an external client, is the sender and receiver of messages, and interacts with the system core through an introduced message middleware client library; The message middleware client library is embedded in message producer and consumer applications. It encapsulates the communication protocol with the system core and provides simple message sending and fetching interfaces for applications. It also has an embedded partition controller and is responsible for communicating with the remote file manager. The file manager is a core server-side component of the system, deployed independently within a service cluster. It manages all message metadata, allocates write locations, performs load balancing, and handles faults. Internally, it contains several key components: Metadata index table: Records the latest file path, offset, status and other information for each message partition or topic, and is maintained and updated by the file manager itself; Partition Controller: Built into the file manager, it is responsible for allocating and managing specific file write node file slots for each message partition; Status probe: Built into the file manager, it proactively and periodically initiates health checks on each file slot to monitor its availability and load status; A file slot is not an independent module, but a logical concept on the distributed file storage layer. It is a file segment that is allocated to a specific message partition on the distributed file storage layer and is currently being written. As the final destination of messages on the disk, it is the smallest unit for performing physical write operations. When a file slot is full, it becomes a read-only archive file, which can be read by consumers or cleaned up by the system according to the recovery retention policy. The distributed file storage layer is the underlying infrastructure of the system, providing the final file persistent storage capability; the file manager maps file slots to the specific physical path of the distributed file storage layer; the file persistence compensation device performs file operations directly on this layer; The file persistence compensation device is a lightweight background process or thread embedded in the file manager, rather than a standalone service. It periodically scans the metadata index table to identify file slots that are being written to but have not been updated for a long time, determining that their original write nodes may have failed. It then performs the finalization operation of the file slot directly on the distributed file storage layer and notifies the file manager to update the metadata index table to ensure data consistency.

8. The system according to claim 7, characterized in that, include: The message producer sends a write request to the file manager through its built-in message middleware client library; After receiving the request, the file manager queries the internal metadata index table to find the file slot currently being written to in the corresponding partition and its node address. Return the valid write address of the file slot to the message middleware client library; After obtaining the write address, the message middleware client library bypasses the file manager and directly writes the message data stream to the specified file slot on the distributed file storage layer; Once the write operation is successfully completed, the message middleware client library notifies the file manager again; The file manager updates its internal metadata index table and returns a confirmation of successful write to the message middleware client library; The message middleware client library returns a confirmation of successful write to the message producer, completing the entire write process.

9. The system according to claim 8, characterized in that, include: When the file manager's status probe detects that a file slot is unresponsive or has timed out during a scheduled health check, the following steps are performed: The file manager marks faulty file slots in the metadata index table as unavailable file slots; The file manager's embedded partition controller will allocate a new file slot for the corresponding partition and update the address of the new file slot in the metadata index table; The file manager returns the new write address to subsequent write requests; Mark unavailable file slots as file slots to be compensated; The file persistence compensation device built into the file manager periodically scans the metadata index table. When it finds a file slot to be compensated, it directly accesses the distributed file storage layer, checks the last state of the file slot, and performs a repair operation to ensure that no data is in a corrupted or unclosed state. Notify the file manager after the compensation is complete; The file manager updates the status of the file slot in the metadata index table to either a closed file slot or an archived file slot; the closed or archived file slot can then be read normally as historical data.

10. The system according to claim 8, characterized in that, include: When a consumer specifies the partition to consume and the starting offset through its built-in message middleware client library, the message middleware client library queries the file manager for the physical file location where the offset message is located; The file manager receives the request, queries the metadata index table, locates the file slot corresponding to the message based on the offset, and obtains the complete access path of the file slot on the distributed file storage layer. Return the complete path to the message middleware client library; After obtaining the file path, the message middleware client library directly reads the file data stream from the specified location on the distributed file storage layer, parses out the specific message content, and returns the parsed message to the consumer.

Images