Message transmission method and device

Abstract

The embodiment of the invention discloses a message transmission method and device, and the method comprises the following steps: carrying out the structure recognition and semantic segmentation of original message data based on a pre-configured partitioning strategy, and generating a plurality of pieces of partitioning data with the structural integrity; generating a metadata record containing a task identifier, a block index, a total block number, a structural hash abstract and a receiving timestamp for each piece of block data; performing data normalization processing on each block of data, wherein the processing comprises the steps of unifying character codes and removing line feed characters and space characters; and sending the data of each block in a transaction mode through a Kafka message queue, and dynamically adjusting the block retry sending time interval and the upper limit of the number of times according to packet loss and delay indexes. According to the embodiment of the invention, the business service can be effectively decoupled, the throughput of the system is remarkably improved, efficient sorting and complete recombination of the block data are realized, and the transmission reliability is further improved.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and specifically relates to a message transmission method and device. Background Art

[0002] As the scale of enterprise business transactions continues to expand, the demand for massive data transmission in scenarios such as real-time monitoring and financial transaction data exchange is growing. Data message transmission is often affected by factors such as network transmission protocol limits on single message size, complex network environments, and the performance of the receiving device.

[0003] Traditional solutions use linked list sorting or linear traversal to transmit data packets. These solutions face multiple bottlenecks under high concurrency, large data volumes, and complex business logic: low transmission efficiency makes it difficult to cope with peak traffic; system modules are tightly coupled, with block strategies tied to transmission logic; data integrity mechanisms are weak, and retry strategies and reassembly algorithms are inefficient, easily leading to data loss or disorder; traditional solutions struggle to adapt to diverse data formats such as JSON, XML, and text, and are unable to dynamically adjust block rules, making them unable to meet the core requirements of modern businesses for efficient, flexible, and reliable big data transmission.

[0004] Application Contents

[0005] The purpose of the embodiments of the present application is to provide a message transmission method and device to solve the defects of the existing technology that cannot achieve transmission efficiency, flexibility and reliability.

[0006] In order to solve the above technical problems, this application is implemented as follows:

[0007] In a first aspect, a message transmission method is provided, which is applied to a sending end, and the method includes the following steps:

[0008] Based on the pre-configured block strategy, the original message data is structurally identified and semantically segmented to generate multiple block data with structural integrity.

[0009] Generate metadata records for each block data, including task ID, block index, total number of blocks, structure hash summary, and receiving timestamp;

[0010] Performing data normalization processing on each block of data, the processing including unifying character encoding and removing line breaks and space characters;

[0011] Each block of data is sent in transaction mode through the Kafka message queue, and the block retry sending time interval and the upper limit are dynamically adjusted according to the packet loss and delay indicators.

[0012] In a second aspect, a message transmission method is provided, which is applied to a receiving end, and the method includes the following steps:

[0013] Receive multiple message block data and metadata transmitted through the Kafka message queue, parse and extract the task identifier, block index and structure hash summary;

[0014] Write each received chunk data into the Redis cache in the key format "task:{task ID}:chunk:{chunk index}" and set an expiration time;

[0015] A skip list structure is constructed based on the block index to manage the sequence information and reception status of each block;

[0016] After the total number of received blocks is met, the message is reassembled according to the sequence information recorded in the jump table, and a structure-level check and content hash comparison are performed;

[0017] If all blocks are not received within the set time window, the corresponding task is marked as an abnormal task and its cache record is cleared.

[0018] In a third aspect, a message transmission device is provided, which is applied to a sending end, and the device includes:

[0019] The recognition module is used to perform structural recognition and semantic segmentation on the original message data based on the pre-configured segmentation strategy, and generate multiple segmented data with structural integrity;

[0020] A generation module is used to generate a metadata record for each block data, including a task identifier, block index, total number of blocks, structure hash summary, and receiving timestamp;

[0021] A processing module, configured to perform data normalization processing on each block of data, wherein the processing includes unifying character encoding and removing line breaks and space characters;

[0022] The sending module is used to send each block of data in transaction mode through the Kafka message queue, and dynamically adjust the block retry sending time interval and the upper limit of the number of times according to the packet loss and delay indicators.

[0023] In a fourth aspect, a message transmission device is provided, which is applied to a receiving end, and the device includes:

[0024] The receiving module is used to receive multiple message block data and metadata transmitted through the Kafka message queue, parse and extract the task identifier, block index and structure hash summary;

[0025] The setup module is used to write each received chunk data into the Redis cache in the key format of "task:{task ID}:chunk:{chunk index}" and set the expiration time;

[0026] A construction module, configured to construct a skip list structure based on the block index to manage the sequence information and reception status of each block;

[0027] The reassembly module is used to reassemble the message according to the sequence information recorded in the jump table after the total number of block receptions is met, and perform structure-level verification and content hash comparison;

[0028] The clearing module is used to mark the corresponding task as an abnormal task and clear its cache record if all blocks are not received within the set time window.

[0029] The embodiment of the present application realizes block processing of asynchronous high-concurrency data packets through block strategy and Kafka message queue, which can effectively decouple business services, significantly improve system throughput, and realize efficient sorting and complete reorganization of block data, thereby improving transmission reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of a message transmission method provided by an embodiment of the present application;

[0031] Figure 2 This is a flow chart of another message transmission method provided in an embodiment of the present application;

[0032] Figure 3 This is a specific implementation diagram of the message transmission method provided in the embodiment of the present application;

[0033] Figure 4 This is a structural diagram of a message transmission device provided in an embodiment of the present application;

[0034] Figure 5 It is a structural diagram of another message transmission device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0036] The present application proposes a packet block transmission and reassembly optimization system based on a skip table index. This system solves three core technical pain points in high-throughput large packet transmission scenarios through an efficient skip table index structure, a flexible block strategy, a reliable transmission mechanism, and optimized resource management:

[0037] 1. Low efficiency of block reorganization

[0038] Traditional solutions use linked list sorting or linear traversal to reassemble blocks, with a time complexity as high as O(N). When the number of blocks is large, the reassembly delay increases significantly, which can easily lead to processing bottlenecks in concurrent scenarios and cannot meet the needs of businesses with high real-time requirements.

[0039] 2. Inefficient block recovery mechanism

[0040] Existing technologies use fixed retransmission strategies (e.g., retrying at fixed intervals), and locating lost blocks relies on sequential traversal, making them extremely inefficient. Due to a lack of block state management, when network jitter causes continuous block loss, traditional solutions can exhaust resources due to frequent retries, or cause messages to time out due to the time it takes to locate lost blocks.

[0041] 3. Systemic risks caused by inadequate resource management

[0042] Traditional solutions lack an expiration mechanism for temporarily storing blocks, which can easily lead to memory overflows. Furthermore, the query efficiency of the block index structure decreases significantly as the amount of data increases.

[0043] Traditional single message transmission solutions have obvious shortcomings: poor architectural flexibility. Due to the business side's strict restrictions on data size, fixed message transmission is difficult to adapt to different business needs; concurrent processing capabilities are limited, upstream and downstream processes are mutually constrained, and performance bottlenecks are prone to occur in high-concurrency scenarios; data transmission stability is insufficient, and the basic retry mechanism is difficult to cope with complex network environments. There is a lack of timeout cleanup and efficient data reorganization methods, which can easily lead to data loss or disorder.

[0044] To address the above issues, the present invention proposes a packet block transmission and reassembly optimization system based on a skip table index, aiming to improve the efficiency, reliability, and flexibility of packet transmission. The system has the following advantages:

[0045] 1. Efficient Reorganization: This system uses a skip list index-based reorganization method with O(logN) time complexity to quickly complete block positioning and sorting. Compared with traditional linked list sorting, it significantly improves efficiency when processing massive data.

[0046] 2. High reliability: The system uses multiple mechanisms to ensure the reliability of message transmission. For example, it uses an exponential backoff algorithm combined with Kafka transactional transmission to ensure reliable transmission; hash checksums and integrity checks ensure the integrity and accuracy of block data.

[0047] 3. Flexible Blocking Strategy: Supports user-defined block rules to adapt to different data types and business needs. At the same time, it has a dynamic block size configuration function that can be adjusted according to actual needs.

[0048] 4. Resource Management Optimization: The temporary storage layer is based on Redis Cluster. Key expiration settings and key format design effectively avoid memory overflow issues and improve data retrieval efficiency. The system also supports a scheduled task cleanup mechanism to further optimize resource management.

[0049] 5. Exception handling mechanism: It has a complete exception handling mechanism that restores data integrity as much as possible through a retry mechanism and abnormal task marking function, and is connected to the abnormal monitoring and alarm platform for manual verification.

[0050] Specifically, the system adopts a layered architecture design, with each layer working together from the bottom layer to the application layer to achieve efficient block transmission and reassembly of messages. The overall system is divided into the following core modules: message block module, jump table index module, transmission control module, resource management module, and reassembly and recovery module.

[0051] 1. Message block module

[0052] Customized block strategy: The system provides a block strategy interface, and users can customize block rules according to business needs, such as block by fixed size, data type or content.

[0053] Block size adjustment configuration: Dynamically adjust the block size as needed based on the configuration functions provided by the system.

[0054] 2. Skip table index module

[0055] Efficient index structure: Uses skip list index instead of traditional linked list or array index, supporting insertion, deletion and search operations with O(logN) time complexity.

[0056] Skip table assisted reassembly: Use the skip table index to quickly locate and sort the received blocks for efficient reassembly.

[0057] 3. Transmission control module

[0058] The system monitors network latency, packet loss rate, and other network status indicators in real time. When a lost segment is detected, it dynamically adjusts the retransmission interval using an exponential backoff algorithm. The first retry interval is set to 1 second, the second to 2 seconds, and so on, with a maximum interval of no more than 30 seconds. The maximum number of retries is 5.

[0059] Data verification and confirmation: Hash verification and integrity verification are used to ensure the accuracy and integrity of block data;

[0060] 4. Resource Management Module

[0061] Distributed temporary storage: Implements distributed caching based on Redis Cluster, supporting high-concurrency access. Key expiration and key format design prevent memory overflow. Resource cleanup mechanism: Scheduled tasks regularly clean up expired or unused block data to free up system resources.

[0062] 5. Exception handling and recovery: Through the retry mechanism and abnormal task marking function, lost or out-of-order blocks can be recovered as much as possible; access to the monitoring and alarm platform to facilitate manual intervention and verification.

[0063] The message transmission method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0064] like Figure 1 FIG. 1 is a flow chart of a message transmission method provided in an embodiment of the present application, which is applied to a sending end and includes the following steps:

[0065] Step 101 : Based on a pre-configured block strategy, the original message data is subjected to structural recognition and semantic segmentation to generate a plurality of block data with structural integrity.

[0066] Among them, the block strategy includes automatically identifying key-value pairs and node level boundaries in JSON or XML structures to avoid the block content destroying the semantic structure.

[0067] Step 102: Generate a metadata record for each block data including a task identifier, a block index, a total number of blocks, a structure hash summary, and a receiving timestamp.

[0068] Step 103 : performing data normalization processing on each block of data, wherein the processing includes unifying character encoding and removing line breaks and space characters.

[0069] Step 104: Send each block of data in transaction mode through the Kafka message queue, and dynamically adjust the block retry sending time interval and the upper limit of the number according to the packet loss and delay indicators.

[0070] In this embodiment, after sending each block of data in transaction mode through the Kafka message queue, a confirmation response from the receiving end can also be received, and the confirmation status is persistently written into the local buffer queue as a basis for subsequent retransmission judgment.

[0071] The embodiment of the present application realizes block processing of asynchronous high-concurrency data packets through block strategy and Kafka message queue, which can effectively decouple business services, significantly improve system throughput, and realize efficient sorting and complete reorganization of block data, thereby improving transmission reliability.

[0072] like Figure 2FIG. 1 is a flow chart of a message transmission method provided in an embodiment of the present application, which is applied to a receiving end. The method includes the following steps:

[0073] Step 201: Receive multiple message block data and metadata transmitted via the Kafka message queue, parse and extract the task identifier, block index and structure hash summary.

[0074] Step 202: Write each received block data into the Redis cache in the key format of "task:{task ID}:chunk:{block index}" and set an expiration time.

[0075] Step 203: construct a skip list structure based on the block index to manage the sequence information and receiving status of each block.

[0076] Among them, the jump table structure contains a receiving timestamp field and a data integrity mark field, which are used for subsequent abnormality positioning and verification comparison.

[0077] Step 204: After the total number of received blocks is met, the message is reassembled according to the sequence information recorded in the jump table, and a structure-level check and content hash comparison are performed.

[0078] Among them, the structural level verification includes performing syntax parsing on the block-reassembled message and verifying the closure of its JSON or XML structure.

[0079] Step 205: If all the blocks are not received within the set time window, the corresponding task is marked as an abnormal task and its cache record is cleared.

[0080] The embodiment of the present application realizes block processing of asynchronous high-concurrency data packets through block strategy and Kafka message queue, which can effectively decouple business services, significantly improve system throughput, and realize efficient sorting and complete reorganization of block data, thereby improving transmission reliability.

[0081] In the embodiment of the present application, a specific implementation process of the message transmission method is as follows: Figure 3 As shown, the following steps are included:

[0082] 1. Initiate message transmission tasks: Users can initiate large message data transmission tasks through the system management platform, API interface, web page, scheduled task scheduling, etc.

[0083] 2. Concurrent processing of requests: The system monitors client upload requests and uses thread pools to efficiently manage concurrent tasks. It also introduces Kafka message queues to buffer request peaks and perform reasonable scheduling and concurrency control of upload requests.

[0084] 3. Receiving data: After receiving the message data, the system first performs syntax parsing, extracts metadata information such as the message data structure, sequence number, hash check code, and verifies the data message, data security signature, and request parameters.

[0085] 4. Chunking strategy: The system implements the ChunkingStrategy interface, identifies messages in different formats such as JSON, XML, and TXT, intelligently schedules the appropriate chunking strategy, and calls built-in chunking rules to perform data message chunking.

[0086] 5. Task construction: Create a chunk upload task record and generate metadata including chunk index (chunkIndex), globally unique identifier, task ID, chunk sequence number, and total number of chunks.

[0087] 6. Data Temporary Storage: Blocked data that passes validation will be stored in the temporary storage layer. The temporary storage layer implements distributed cache storage based on RedisCluster. The system sets a key expiration time for each block of data, and the key format is designed using the "task:taskId:chunk:{chunkIndex}" key format. "task" represents the task type identifier, "taskId" is the globally unique identifier of the task, "chunk" represents the chunk, and "chunkIndex" is the chunk sequence number. This key format facilitates rapid location and retrieval of chunked data, improving data retrieval efficiency while effectively avoiding memory overflow issues.

[0088] 7. Message Reassembly: Count the total number of blocks, compare message lengths, and verify the number of blocks. Once the reassembly conditions are met, aggregate blocks by task ID: Aggregate the block data belonging to the same task. Use skip list index reassembly to quickly sort the aggregated block data and complete message reassembly.

[0089] 8. Timeout handling: The system sets a reasonable timeout for each upload task. It issues an alarm, logs the task, and temporarily clears the data blocks that are not fully received due to the timeout to avoid resource occupation.

[0090] 9. Reliable transmission mechanism: An exponential backoff algorithm combined with Kafka transactional transmission is used to ensure reliable transmission of blocks. The first retry interval is set to 1 second, the second retry interval is 2 seconds, and the maximum interval does not exceed 30 seconds. The maximum number of retries is 5.

[0091] 10. System monitoring: Integrate Promothus to collect transmission indicators, monitor thread pool, Kafka consumption rate, and support manual re-initiation of transmission in case of abnormality.

[0092] 11. Data delivery: After block transmission and reassembly are completed, the system will encapsulate the reassembled message, convert it according to the data format and transmission protocol required by the downstream business module or recipient, and deliver it to the downstream system for subsequent business processing.

[0093] It can be seen that the message transmission method in the embodiment of the present application includes: the user initiates a request to upload large message data in blocks; the system receives the request and processes the block data in parallel, and reorganizes the block data according to the block index; after the system completes the message reorganization, it passes the reorganized message to the user business system in accordance with the agreed interface specifications and data format for subsequent business logic processing.

[0094] The embodiments of the present application focus on building a modular layered architecture based on domain-driven design, clarifying the functional definitions and interface boundaries of each layer and the interaction protocols, achieving high cohesion and low coupling of the system, and ensuring the scalability of the system. Among them, through customized block strategies, a flexible and scalable block strategy interface is provided, supporting multiple data formats such as JSON, XML, TXT, etc.; through highly reliable data transmission and reorganization mechanisms, the kafka message queue is introduced to decouple upstream and downstream processes and achieve load balancing under high concurrency. The O(log n) time complexity sorting of block data is achieved through the skip table index structure to ensure the orderly reorganization of data in a distributed environment; through real-time monitoring and dynamic adjustment, the Prometheus monitoring and alarm system is integrated, transmission indicators are collected in real time, key parameters such as the thread pool and the kafka consumption rate are monitored, and dynamic adjustment of data block size, thread pool parameter configuration, etc. according to monitoring data is supported to achieve intelligent management and optimization of the system.

[0095] Compared with the prior art, the embodiments of the present application have the following advantages:

[0096] 1. Flexible expansion

[0097] Abstract the block strategy interface to implement the data block algorithm, and the data message format is highly flexible and extensible.

[0098] 2. Strong concurrent processing capabilities

[0099] Use the Kafka message queue in combination with the thread pool to implement asynchronous high-concurrency data message block processing, effectively decouple business services, significantly improve system throughput, and be able to cope with high-concurrency and large-data volume business scenarios.

[0100] 3. Stronger data integrity protection

[0101] The exponential backoff retry algorithm is combined with the skip table index reorganization algorithm to achieve efficient sorting and complete reorganization of block data, ensuring zero data loss and zero confusion.

[0102] 4. Intelligent monitoring and optimization

[0103] Integrate a real-time monitoring system to monitor key indicators and dynamically adjust block strategies and thread pool parameters.

[0104] like Figure 4 FIG. 1 is a schematic diagram of a structure of a message transmission device provided in an embodiment of the present application, which is applied to a sending end and includes:

[0105] The identification module 410 is used to perform structural identification and semantic segmentation on the original message data based on a pre-configured segmentation strategy, and generate a plurality of segmented data with structural integrity.

[0106] Among them, the block strategy includes automatically identifying key-value pairs and node level boundaries in JSON or XML structures to avoid the block content destroying the semantic structure.

[0107] The generation module 420 is used to generate a metadata record containing a task identifier, a block index, a total number of blocks, a structure hash summary and a receiving timestamp for each block data.

[0108] The processing module 430 is configured to perform data normalization processing on each block of data, wherein the processing includes unifying character encoding and removing line breaks and space characters.

[0109] The sending module 440 is used to send each block of data in a transaction mode through the Kafka message queue, and dynamically adjust the block retry sending time interval and the upper limit of the number of times according to the packet loss and delay indicators.

[0110] Furthermore, the above device further includes:

[0111] The writing module is used to receive the confirmation response from the receiving end and write the confirmation status persistently into the local buffer queue as a basis for subsequent retransmission judgment.

[0112] The embodiment of the present application realizes block processing of asynchronous high-concurrency data packets through block strategy and Kafka message queue, which can effectively decouple business services, significantly improve system throughput, and realize efficient sorting and complete reorganization of block data, thereby improving transmission reliability.

[0113] like Figure 5 FIG. 1 is a schematic diagram of the structure of another message transmission device provided in an embodiment of the present application, which is applied to a receiving end and includes:

[0114] The receiving module 510 is used to receive multiple message block data and metadata transmitted via the Kafka message queue, parse and extract the task identifier, block index and structure hash summary.

[0115] The setting module 520 is used to write each received block data into the Redis cache in the key format of "task:{task identifier}:chunk:{block index}" and set an expiration time.

[0116] The construction module 530 is used to construct a skip list structure based on the block index to manage the sequence information and reception status of each block.

[0117] Among them, the jump table structure contains a receiving timestamp field and a data integrity mark field, which are used for subsequent abnormality positioning and verification comparison.

[0118] The reassembly module 540 is used to reassemble the message according to the sequence information recorded in the jump table after the total number of block receptions is met, and perform structure-level verification and content hash comparison.

[0119] Among them, the structural level verification includes performing syntax parsing on the block-reassembled message and verifying the closure of its JSON or XML structure.

[0120] The clearing module 550 is configured to mark the corresponding task as an abnormal task and clear its cache record if all the blocks are not received within a set time window.

[0121] The embodiment of the present application realizes block processing of asynchronous high-concurrency data packets through block strategy and Kafka message queue, which can effectively decouple business services, significantly improve system throughput, and realize efficient sorting and complete reorganization of block data, thereby improving transmission reliability.

[0122] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the various processes of the above-mentioned message transmission method embodiment and can achieve the same technical effect. To avoid repetition, the details are not described here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0123] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0125] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A message transmission method, characterized in that: Applied to the sending end, the method includes the following steps: Based on the pre-configured block strategy, the original message data is structurally identified and semantically segmented to generate multiple block data with structural integrity. Generate metadata records for each block data, including task ID, block index, total number of blocks, structure hash summary, and receiving timestamp; Performing data normalization processing on each block of data, the processing including unifying character encoding and removing line breaks and space characters; Each block of data is sent in transaction mode through the Kafka message queue, and the block retry sending time interval and the upper limit are dynamically adjusted according to the packet loss and delay indicators.

2. The method according to claim 1, characterized in that The chunking strategy includes automatically identifying key-value pairs and node level boundaries in JSON or XML structures to avoid chunking content from destroying the semantic structure.

3. The method according to claim 1, characterized in that After sending each block of data in transaction mode through the Kafka message queue, the following steps are also included: Receive the confirmation response from the receiving end and write the confirmation status persistently to the local buffer queue as the basis for subsequent retransmission judgment.

4. A message transmission method, characterized in that: Applied to the receiving end, the method comprises the following steps: Receive multiple message block data and metadata transmitted through the Kafka message queue, parse and extract the task identifier, block index and structure hash summary; Write each received chunk data into the Redis cache in the key format "task:{task ID}:chunk:{chunk index}" and set an expiration time; A skip list structure is constructed based on the block index to manage the sequence information and reception status of each block; After the total number of received blocks is met, the message is reassembled according to the sequence information recorded in the jump table, and a structure-level check and content hash comparison are performed; If all blocks are not received within the set time window, the corresponding task is marked as an abnormal task and its cache record is cleared.

5. The method according to claim 4, characterized in that The jump table structure includes a receiving timestamp field and a data integrity mark field, which are used for subsequent abnormality location and verification comparison; the structural level verification includes performing syntax parsing on the block-reassembled message and verifying its JSON or XML structure closure.

6. A message transmission device, characterized in that: Applied to a transmitting end, the device includes: The recognition module is used to perform structural recognition and semantic segmentation on the original message data based on the pre-configured segmentation strategy, and generate multiple segmented data with structural integrity; A generation module is used to generate a metadata record for each block data, including a task identifier, block index, total number of blocks, structure hash summary, and receiving timestamp; A processing module, configured to perform data normalization processing on each block of data, wherein the processing includes unifying character encoding and removing line breaks and space characters; The sending module is used to send each block of data in transaction mode through the Kafka message queue, and dynamically adjust the block retry sending time interval and the upper limit of the number of times according to the packet loss and delay indicators.

7. The device according to claim 6, characterized in that The chunking strategy includes automatically identifying key-value pairs and node level boundaries in JSON or XML structures to avoid chunking content from destroying the semantic structure.

8. The device according to claim 6, characterized in that Also includes: The writing module is used to receive the confirmation response from the receiving end and write the confirmation status persistently into the local buffer queue as a basis for subsequent retransmission judgment.

9. A message transmission device, characterized in that: Applied to a receiving end, the device includes: The receiving module is used to receive multiple message block data and metadata transmitted through the Kafka message queue, parse and extract the task identifier, block index and structure hash summary; The setup module is used to write each received chunk data into the Redis cache in the key format "task:{task ID}:chunk:{chunk index}" and set the expiration time; A construction module, configured to construct a skip list structure based on the block index to manage the sequence information and reception status of each block; The reassembly module is used to reassemble the message according to the sequence information recorded in the jump table after the total number of block receptions is met, and perform structure-level verification and content hash comparison; The clearing module is used to mark the corresponding task as an abnormal task and clear its cache record if all blocks are not received within the set time window.

10. The device according to claim 9, characterized in that The jump table structure includes a receiving timestamp field and a data integrity mark field, which are used for subsequent abnormality location and verification comparison; the structural level verification includes performing syntax parsing on the block-reassembled message and verifying its JSON or XML structure closure.

Images