Data transmission method and system based on dynamic routing and intelligent storage scheduling

Abstract

The embodiment of the invention relates to the technical field of data processing and transmission, and provides a data transmission method and system based on dynamic routing and intelligent storage scheduling, and the system comprises a Kafka data input layer which is used for receiving a to-be-transmitted message queue inputted by a user; the transmission service module is used for analyzing the message queue and delivering the message queue to a target queue according to the selected dynamic route; the storage scheduling engine is used for carrying out storage resource allocation on the target queue according to the residual capacity of the storage equipment, and scheduling the storage resource position of the target queue for the health degree prediction data of the storage equipment based on a prediction model; the physical storage pool is used for storing the target queue, and the physical storage pool is composed of the storage devices. Therefore, the message delivery delay is remarkably reduced, and breakthrough improvement is achieved on the two key indexes of average delay and high-load extreme delay.

Description

Technical Field

[0001] This invention relates to the field of data processing and transmission technology, and in particular to a data transmission method and system based on dynamic routing and intelligent storage scheduling. Background Technology

[0002] Message Queue Middleware (IBM MQ) is an enterprise-grade message middleware primarily used to solve the problem of reliable, asynchronous communication between different systems. Through a "message passing" mechanism, it enables applications, services, or heterogeneous systems (such as different operating systems and programming languages) to exchange data securely and efficiently, making it particularly suitable for distributed architectures and mission-critical scenarios.

[0003] However, existing MQ message transmission technology has the following problems: (1) Rigid routing: Unable to dynamically switch paths based on application status (such as service downtime); (2) Storage resource mismatch: Dead letters / logs are stored uniformly without being stratified according to data value; (3) Poor scalability: Virtual nodes require manual configuration of physical mapping, and cross-node migration requires downtime. Summary of the Invention

[0004] This invention provides a data transmission method and system based on dynamic routing and intelligent storage scheduling to solve the defects of poor dynamic routing adaptability, poor scalability and storage resource mismatch in message delivery in hybrid cloud-edge environments in the prior art. It achieves a significant reduction in message delivery latency and makes breakthrough improvements in two key indicators: average latency and extreme latency under high load.

[0005] This invention provides a data transmission system based on dynamic routing and intelligent storage scheduling, comprising: The Kafka data input layer is used to receive user-input message queues that are yet to be transmitted. The transmission service module is used to parse the message queue and deliver the message queue to the target queue according to the selected dynamic route; A storage scheduling engine is used to allocate storage resources to the target queue based on the remaining capacity of the storage device, and to schedule the storage resource location of the target queue based on the health prediction data of the storage device using a prediction model. A physical storage pool is used to store the target queue, and the physical storage pool is composed of the storage device.

[0006] Optionally, the transmission service module includes: The configuration management submodule is used to load, parse, and maintain the global configuration information of the data transmission system, and load the configuration file when the data transmission system starts to initialize the global structure; The message processing submodule is used to parse the message queue and extract features from the target queue; The message routing submodule is used to dynamically select the message queue delivery path based on the parsed data content, application device, and configuration rules. The message delivery submodule is used to deliver the parsed message queue to the target queue via the data transmission bus according to the selected message queue delivery path; The log submodule is used to record key events generated after message queue delivery to obtain logs. The key events include delivery success, delivery failure, error recovery, and configuration loading information. The log submodule classifies the logs based on their content and access frequency, and distributes the classified logs to the corresponding storage layer through a dynamic scheduler. The content of the logs includes at least one of timestamp, time type, and priority. The file management submodule is used to manage the files required during the operation of the data transmission system. The management operations include configuration loading, persistent storage, log rotation, and cross-platform adaptation. The persistent storage includes fault-domain-aware persistence, used to execute replica distribution strategies and cross-platform file operations. The replica distribution strategy includes data type strategies based on configuration information, unconfirmed private messages, and the number of replicas; and fault domain isolation level strategies based on server-level, rack-level, and data center-level fault domains. The cross-platform file operations include support for atomic write operations and log rotation threshold adjustment.

[0007] Optionally, the storage scheduling engine includes: A state sensor is used to continuously monitor the middleware status and physical metrics, wherein the middleware status includes the length of the transmission message queue, task execution status, and resource usage, and the physical metrics include the remaining capacity, read / write performance, and failure rate of the storage device. The strategy decision-maker is used to make storage strategy decisions based on the information monitored by the state sensor, the pre-set storage strategy, and business requirements. A predictive model is used to predict the future health of the storage device using historical data and current system status information; The coordinator is used to convert the storage policy formulated by the policy decision-maker into specific operation instructions and coordinate the execution of the corresponding operation instructions by each sub-module.

[0008] Optionally, the physical storage pool is divided into a four-layer structure based on access characteristics; The four-layer structure, from top to bottom, consists of edge storage media, central storage cluster, nearline storage nodes, and archive storage facilities. Each layer of storage media is connected to the storage scheduling engine through a unified interface.

[0009] This invention also provides a data transmission method based on dynamic routing and intelligent storage scheduling, characterized in that it is applied to a data transmission system and includes: Initialize the data transmission system and receive the message queue to be transmitted from the user input; The message queue is parsed, and the message queue is delivered to the target queue according to the selected dynamic route; Obtain the remaining capacity of the storage device and allocate storage resources to the target queue based on the remaining capacity of the storage device; Based on the health prediction data of the storage device using a prediction model, the storage resource location of the target queue is scheduled; The target queue is stored based on the location of the scheduled storage resources.

[0010] In one possible implementation, the method further includes: The message queue is parsed to obtain the data content; Based on the data content, application device, and configuration rules, dynamically select the message queue delivery path; Based on the selected message queue delivery path, the parsed message queue is delivered to the target queue via the data transmission bus.

[0011] In one possible implementation, the method further includes: Record key events generated after message queue delivery to obtain logs. The key events include delivery success, delivery failure, error recovery, and configuration loading information. The logs are classified based on their content and access frequency, and the classified logs are distributed to the corresponding storage layer through a dynamic scheduler. The content of the logs includes at least one of timestamp, time type, and priority.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data transmission method as described above.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data transmission method as described above.

[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data transmission method as described above.

[0015] This invention provides a data transmission method and system based on dynamic routing and intelligent storage scheduling, comprising a Kafka data input layer for receiving user-input message queues to be transmitted; a transmission service module for parsing the message queues and delivering them to target queues according to selected dynamic routes; a storage scheduling engine for allocating storage resources to the target queues based on the remaining capacity of storage devices and scheduling the storage resource locations of the target queues based on health prediction data of the storage devices using a prediction model; and a physical storage pool for storing the target queues, wherein the physical storage pool is composed of the storage devices. Compared to the shortcomings of existing technologies in message delivery in hybrid cloud-edge environments, such as poor adaptability and scalability of dynamic routing and storage resource mismatch, this solution significantly reduces message delivery latency and achieves breakthrough improvements in both average latency and high-load extreme latency.

[0016] This invention provides a data transmission method and system based on dynamic routing and intelligent storage scheduling. The method involves initializing the data transmission system and receiving a message queue to be transmitted from a user; parsing the message queue and delivering it to a target queue according to a selected dynamic route; obtaining the remaining capacity of the storage device and allocating storage resources to the target queue based on that capacity; scheduling the storage resource location of the target queue based on a predictive model's prediction of the storage device's health; and storing the target queue based on the scheduled storage resource location. Compared to the shortcomings of existing technologies in message delivery under hybrid cloud-edge environments, such as poor adaptability, poor scalability, and storage resource mismatch due to dynamic routing, this solution significantly reduces message delivery latency through a dynamic routing and intelligent storage collaborative mechanism, significantly improves storage efficiency through an intelligent hierarchical compression strategy, achieves a revolutionary breakthrough through a cross-fault domain replica strategy, and effectively suppresses system performance jitter through predictive migration technology. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the architecture of the data transmission system based on dynamic routing and intelligent storage scheduling provided by the present invention.

[0019] Figure 2 This is a storage scheduling flowchart of the data transmission system based on dynamic routing and intelligent storage scheduling provided by the present invention.

[0020] Figure 3 This is a schematic diagram of fault domain-aware storage in the data transmission system based on dynamic routing and intelligent storage scheduling provided by the present invention.

[0021] Figure 4 This is a flowchart of the data transmission method based on dynamic routing and intelligent storage scheduling provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0025] Figure 1 This is a schematic diagram of the architecture of the data transmission system based on dynamic routing and intelligent storage scheduling provided by the present invention, as shown below. Figure 1 As shown, this system includes a Kafka data input layer, which is used to receive user-input message queues to be transmitted. The transmission service module is used to parse the message queue and deliver the message queue to the target queue according to the selected dynamic route; A storage scheduling engine is used to allocate storage resources to the target queue based on the remaining capacity of the storage device, and to schedule the storage resource location of the target queue based on the health prediction data of the storage device using a prediction model. A physical storage pool is used to store the target queue, and the physical storage pool is composed of the storage device.

[0026] In this embodiment of the invention, the transmission service module is deployed on a dedicated MQ communication server. This module is designed as a high-performance process resident in memory, with clear baseline requirements for underlying hardware resources, including multi-core processors and large-capacity memory configurations. Simultaneously, it strictly controls the resource usage ratio of the system process itself to ensure stable service capabilities under complex business scenarios. Specifically, this includes: The configuration management submodule is used to load, parse, and maintain the global configuration information of the data transmission system. After the data transmission system starts, it loads and reads the configuration file, parses the configuration file, and performs the global structure initialization process. In this embodiment of the invention, the initialization process includes: loading the site dictionary from the disk, parsing the address mapping relationship, and performing the storage policy engine initialization process. The address format is: {region code}-{rack number}-{device number}, and the code length is 3-5 characters. The network latency threshold for the initialization process is set to 50-200ms; if the threshold is exceeded, degradation is triggered.

[0027] The message processing submodule is used to parse received MQ messages and extract features from the target queue. The message routing submodule is used to dynamically select the message delivery path based on the content of the parsed MQ message, the application device, and the configuration rules. The message delivery submodule is used to send the parsed MQ message to the target queue via the data transmission bus according to the selected delivery path; The logging submodule is used to record key events that occur after message delivery, including delivery success / failure, error recovery, and configuration loading information; The file management submodule is used for file operations required during the operation of the data transmission system, including configuration loading, persistent storage, log rotation, and cross-platform adaptation.

[0028] Figure 3 This is a schematic diagram of fault domain-aware storage in a data transmission system based on dynamic routing and intelligent storage scheduling provided by the present invention. In this embodiment, the persistent storage in the file management submodule includes fault domain-aware persistence, used to execute replica distribution strategies and cross-platform file operations. The replica distribution strategy includes data type strategies based on configuration information, unconfirmed private messages, and the number of replicas; and fault domain isolation level strategies based on server-level, rack-level, and data center-level. The cross-platform file operations include atomic write operation support (xfs / ext4 (Linux), NTFS (Windows)) and log rotation threshold adjustment, wherein the log rotation threshold is 1-5GB. Figure 3 As shown, two copies of the same data (such as a payment message) are distributed in different fault domains (rack 1 and rack 3). No copy is stored in rack 2 (to avoid the risk of a single point of failure). The black dashed box is the boundary of the physical fault domain (such as the power module / network switch isolation area).

[0029] It should be noted that the intermediate products in the embodiments of the present invention include classification tag messages and storage metadata indexes. The classification tag messages include extended fields in the original message header, including the initial storage level, compression identifier, and encryption identifier. The storage metadata indexes include the mapping between LevelDB database record data IDs and physical locations.

[0030] Figure 2 This is a storage scheduling flowchart of the data transmission system based on dynamic routing and intelligent storage scheduling provided by the present invention, as shown below. Figure 1 and Figure 2 As shown, the storage scheduling engine includes: A state sensor continuously monitors middleware status and physical metrics. Middleware status includes data transfer queue length, task execution status, and resource usage. Physical metrics cover storage device remaining capacity, read / write performance, and failure rate. By collecting this information in real time, the storage scheduling engine receives comprehensive and accurate system operation status data, enabling it to understand the current system condition and providing foundational data support for subsequent decision-making.

[0031] The strategy decision-maker is used to make storage strategy decisions based on information obtained from the state sensor, combined with pre-set storage strategies and business requirements. In this embodiment of the invention, for example, when it is detected that the remaining capacity of a certain storage device is insufficient, the strategy decision-maker will decide whether to perform data migration, adjust storage resource allocation, or trigger a new storage resource request based on the importance and usage frequency of the data.

[0032] A predictive model is used to predict future data storage needs, system load changes, and storage device failure probabilities using historical data and current system status information. In embodiments of the present invention, for example, by analyzing data growth trends in historical data, the storage capacity demand for a future period of time is predicted; and based on device operating parameters and fault history records, the possible time points when the device may fail are predicted.

[0033] The execution coordinator is used to translate the storage policy formulated by the policy decision-maker into specific operation instructions and coordinate the execution of the corresponding operation instructions by each sub-module. For example, when data migration is required, the execution coordinator will send a new transmission path instruction to the message routing module and a target storage location instruction to the message delivery module to ensure that each module works together to accurately complete the execution of the storage policy.

[0034] The physical storage resource pool in this embodiment of the invention adopts a layered architecture to adapt to different performance and cost requirements. The resource pool is divided into a four-layer structure according to data access characteristics: from top to bottom, it consists of edge high-speed storage media providing extreme responsiveness and targeting low-latency scenarios, a central high-performance storage cluster, large-capacity nearline storage nodes, and economical archive storage facilities. The top layer, edge high-speed storage media targeting low-latency scenarios, provides extreme responsiveness; each layer defines mainstream device types and performance characteristics, and is equipped with alternative technical solutions to enhance deployment flexibility. Each layer of storage media connects to the storage scheduling engine through a unified interface, forming a physical infrastructure that supports dynamic routing decisions.

[0035] In this embodiment of the invention, the policy configuration file includes: (1) storage_tiers.conf supports JSON / YAML formats; (2) The compression algorithm can be replaced with lz4 (low CPU overhead), zstd (high compression ratio), or brotli (web log optimization); (3) The encryption module supports software implementation or hardware acceleration card.

[0036] In this embodiment of the invention, the data transmission system based on dynamic routing and intelligent storage scheduling employs message classification and storage decision-making. The message classification rules comprehensively consider multiple factors such as business attributes, load characteristics, and technical indicators to assign initial storage levels and processing strategies to different types of messages. For example, high-priority business messages trigger high-speed storage paths based on specific identifiers, large-volume log data is matched with capacity optimization strategies, and abnormal messages arising from system failures are handled using security-oriented methods. Each type of message is associated with a differentiated combination of compression algorithms and encryption schemes to ensure a balance between storage efficiency and security requirements. In storage decision-making, a dynamic parameter tuning mechanism is introduced to enhance environmental adaptability. The heat prediction model uses a configurable smoothing coefficient range, supporting flexible adjustment of trend sensitivity based on business fluctuation characteristics. The storage migration strategy sets dual trigger conditions, intelligently determining based on the historical trend and duration of storage resource utilization. These two sets of core parameters constitute a flexible decision-making framework, enabling the system to dynamically adjust the strategy strength within a preset range based on real-time load conditions, avoiding resource mismatch or response lag caused by fixed thresholds.

[0037] To more clearly illustrate the content of the embodiments of the present invention, two application scenarios are established, including a railway signal dead-letter intelligent acceleration scenario and an audit log layered compression scenario.

[0038] The intelligent acceleration scenario for dead letters in railway signaling was specifically tested in a high-priority dead letter processing scenario in an edge environment. The aim was to evaluate the acceleration effect of the dynamic routing and intelligent storage collaborative mechanism on critical business messages. The experiment used dead letter samples generated by a real railway signaling system, and a test environment was built on a professional edge computing platform. The server was equipped with a high-performance domestic processor and large-capacity memory, and NVMe solid-state storage and traditional hard disk drives were configured as control group storage media to construct a complete performance comparison benchmark.

[0039] The experiment rigorously simulated real-world fault scenarios: after manually triggering a signal service interruption and generating a dead letter, the system's rule engine immediately identified message attributes and forcibly allocated the dead letter to the edge high-speed storage layer according to a preset strategy. The execution phase employed a high-strength compression algorithm and hardware-accelerated encryption technology in tandem, ensuring data integrity through atomic write operations. Performance measurement utilized industry-standard testing tools, collecting key metrics under simulated high-concurrency conditions.

[0040] The results demonstrate that the method of this invention achieves a breakthrough performance improvement: compared with traditional mechanical hard disk methods, write latency is reduced to a negligible level, and read response time reaches sub-millisecond levels. Computational resource consumption during processing is significantly optimized, with CPU utilization decreasing by several times. The most critical business metric—dead letter redelivery timeliness—is improved from a timeout and unavailable state to sub-second completion, meeting the stringent millisecond response requirements of railway signaling systems. This confirms the core value of intelligent storage decision-making in edge-critical business scenarios.

[0041] This experiment, focusing on layered compression of audit logs for optimizing massive audit log storage, verifies the breakthrough value of customized compression algorithms in cost control. The experiment uses real bank audit log samples, with data formats conforming to international financial standards. The test platform integrates high-performance servers and dedicated hardware acceleration devices, and a rigorous evaluation system is constructed by comparing mainstream compression solutions.

[0042] This invention was tested on a dedicated railway testing platform, and verified based on railway operation data. The hardware platform uses a high-performance server (64-core domestic processor, 512GB memory) and a dedicated hardware acceleration card. The storage system is configured with a tiered storage architecture, with the hot data layer consisting of NVMe SSDs (4TB, PCIe 4.0) and the cold data layer consisting of enterprise-grade HDDs (20TB, RAID-6 array). Data samples were collected from real railway audit logs.

[0043] In the process of log classification and hierarchical storage, a log classification algorithm is developed to hierarchically classify logs based on log content (timestamp, event type, priority) and access frequency. High-frequency access logs (such as real-time signal logs) are allocated to the hot data layer, while low-frequency access logs (such as historical scheduling records) are archived to the cold data layer. This achieves automatic log classification. A dynamic scheduler then distributes the classified logs to the corresponding storage layer. The hot data layer uses high-performance NVMe SSDs, while the cold data layer uses high-capacity HDDs.

[0044] A customized zstd compression pipeline was developed, training a dedicated zstd compression dictionary based on railway audit log samples to optimize compression efficiency for specific patterns (such as repeated timestamps and signal IDs). zstd algorithm patches were developed to enhance compression capabilities for mixed data (structured + unstructured) in railway logs and improve performance in the entropy coding stage. A dedicated API was designed to seamlessly integrate the zstd compression engine with hardware acceleration cards, reducing CPU intervention through DMA (Direct Memory Access) technology and increasing compression speed to 15GB / s. A multi-stage compression pipeline was constructed, including preprocessing (data chunking), compression (parallel zstd threads), and verification (CRC32), ensuring high throughput and data integrity.

[0045] In this embodiment of the invention, firstly, a dynamic routing mechanism perceives the status of network nodes in real time, automatically avoiding faulty or congested nodes within milliseconds, ensuring that the message path is always in an optimal state. Secondly, a tiered storage strategy implements differentiated processing for high-frequency messages, compressing the physical access latency of critical data to the microsecond level through the direct write capability of edge high-speed storage media. The synergistic effect of these two mechanisms fundamentally solves the latency accumulation problem caused by path rigidity and storage performance bottlenecks in traditional solutions, providing stable and reliable message transmission guarantees for industrial scenarios with stringent real-time requirements.

[0046] The optimization effect of this invention stems from a rigorous space-saving theoretical model. Through the synergy of a tiered storage strategy and advanced compression technology, the system compresses unit storage costs to levels far below industry benchmarks. Particularly for massive archived data, the economical compression strategy implemented in conjunction with object storage characteristics achieves a significant reduction in unit costs while ensuring data accessibility, resulting in a quantifiable storage economic advantage. This mechanism fundamentally changes the mismatch between resource consumption and business value in traditional storage solutions.

[0047] It should be noted that in resource-intensive embedded device environments, this invention achieves a breakthrough in solving the storage performance bottleneck in edge computing scenarios by employing lightweight and transmission optimization technologies. The experiment used mainstream edge hardware platforms and public network communication environments to simulate the operational constraints of real IoT nodes. Compared to traditional solutions, the technical architecture of this invention achieves significant optimizations in three major dimensions: log processing latency, storage space usage, and computational load.

[0048] Core technology adaptation is reflected in dual innovation: (1) Lightweight compression engine: Low-overhead compression algorithm is selected to keep CPU usage at a very low level while ensuring a reasonable compression ratio, so as to avoid service interruption due to computational overload of edge devices; (2) Intelligent Differential Persistence: Breaking through the traditional full-write mode, it significantly reduces the actual amount of data written by capturing change. This mechanism only synchronizes data difference fragments and combines sliding window verification to ensure data integrity, fundamentally solving the dual challenges of bandwidth fluctuations and limited storage space.

[0049] According to the objectives of the embodiments of the present invention, the present invention also provides a data transmission method based on dynamic routing and intelligent storage scheduling. Figure 4 This is a flowchart illustrating the data transmission method provided by the present invention, as shown below. Figure 4 As shown, the method includes the following: S41. Initialize the data transmission system and receive the message queue to be transmitted from the user input.

[0050] During the initialization phase, the file management submodule loads the site dictionary, routing table, storage strategy, and other configurations into memory at once and verifies their validity. Simultaneously, the configuration management submodule establishes a connection pool with the Kafka input layer and sets the consumption point, concurrency level, and degradation threshold. When a user submits a "message queue to be transmitted" via API or console, the system first verifies the Topic name, message format, and schema version. If the verification passes, a globally unique batch-ID is assigned to the batch of messages and written to the local write-ahead log, ensuring that subsequent steps can be recovered via batch-ID replay even in the event of a system crash.

[0051] S42. The message queue is parsed and delivered to the target queue according to the selected dynamic route.

[0052] The message queue is parsed to obtain the data content; based on the data content, application device, and configuration rules, the message queue delivery path is dynamically selected; and the parsed message queue is delivered to the target queue via the data transmission bus according to the selected message queue delivery path.

[0053] Furthermore, key events generated after message queue delivery are recorded to obtain logs, where key events include delivery success, delivery failure, error recovery, and configuration loading information; the logs are classified based on their content and access frequency, and the classified logs are distributed to the corresponding storage layer through a dynamic scheduler, where the log content includes at least one of timestamp, time type, and priority.

[0054] Specifically, the message processing submodule parses the message in three parts: protocol header, message body, and extended fields, extracting features such as business keys, message priority, and target application identifiers. The message routing submodule inputs these features into the rule engine, which scores the data based on real-time collected link latency, queue backlog, and node health, calculating the optimal delivery path within 50 ms. If the path is abnormal, a fallback strategy is immediately triggered to switch to a backup channel. After obtaining the path, the message delivery submodule writes messages in batches to the target queue via the built-in data transmission bus (based on the zero-copy Netty framework) and returns confirmation information. Simultaneously, the logging submodule writes key events such as "delivery success / failure, error code, retry count, and configuration version number" into a circular buffer in structured JSON format. The background asynchronous thread tags logs according to three dimensions: "timestamp, event type, and priority." High-frequency hot logs are directly sent to the NVMe SSD hot layer, while cold logs are compressed using zstd and sent to the HDD cold layer. The entire classification process is automatically completed by the dynamic scheduler without manual intervention.

[0055] S43. Obtain the remaining capacity of the storage device and allocate storage resources to the target queue according to the remaining capacity of the storage device.

[0056] The status sensor can report the remaining capacity, IOPS, and bandwidth utilization of all storage devices every 5 seconds; the policy decision-maker substitutes the capacity information into a linear programming model, and calculates the initial placement scheme that makes the overall remaining capacity of the cluster most balanced under the constraints of replica number, fault domain isolation, and read-write ratio, and generates a "partition-device" mapping table; the execution coordinator distributes the mapping table to each node, and the nodes create the corresponding data directory and index file according to the table to complete the pre-allocation of resources.

[0057] S44. Based on the health prediction data of the storage device obtained by the prediction model, schedule the storage resource location of the target queue.

[0058] The predictive model can use historical data from the past 7 days as a training set. Input features include read / write operations per second, disk SMART metrics, temperature, network retransmission rate, and node failure records. It outputs a health score for the next 24 hours and the expected failure time. If the health of a device falls below the threshold or is predicted to fail within 4 hours, the policy decision-maker immediately triggers the "early migration" process: an incremental snapshot is generated for the data already written to the target queue, and the snapshot is copied to a new device with high health using multi-threaded concurrency. After the copy is completed, the metadata index is updated. The entire process is transparent to the upper-layer business. During the migration, read / write requests still fall on the original device. Once the replica consistency verification is passed, the route is switched instantly, achieving "hot migration with zero interruption".

[0059] S45. Store the target queue based on the location of the scheduled storage resources.

[0060] The physical storage pool is tiered based on performance and cost. The hottest data is placed on edge NVMe cards, warm data on a central SSD cluster, cold data on large-capacity HDDs, and extremely cold data is archived to object storage after high compression. Each layer uses a unified block interface and object interface encapsulation, so the routing layer sees the same set of "logical addresses" regardless of where the data ultimately lands. Once all messages in the target queue have been written, verified, and replicated, the system submits the consumption position to Kafka and returns a "transmission complete" callback to the user. This marks the end of the entire lifecycle of a message, from access, routing, scheduling to persistence.

[0061] This invention provides a data transmission method based on dynamic routing and intelligent storage scheduling. The method initializes a data transmission system and receives a message queue to be transmitted from a user. It then parses the message queue and delivers it to a target queue according to a selected dynamic route. The remaining capacity of the storage device is obtained, and storage resources are allocated to the target queue based on this capacity. The storage resource location of the target queue is scheduled based on a predictive model's prediction of the storage device's health. Finally, the target queue is stored based on the scheduled storage resource location. Compared to existing technologies that suffer from poor adaptability, poor scalability, and storage resource mismatch in message delivery within a hybrid cloud-edge environment, this method significantly reduces message delivery latency through a dynamic routing and intelligent storage collaboration mechanism, significantly improves storage efficiency through an intelligent hierarchical compression strategy, achieves a revolutionary breakthrough through a cross-fault domain replica strategy, and effectively suppresses system performance jitter through its predictive migration technology.

[0062] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a data transmission method based on dynamic routing and intelligent storage scheduling. This method includes: initializing the data transmission system and receiving a message queue to be transmitted input by the user; parsing the message queue and delivering the message queue to a target queue according to the selected dynamic route; obtaining the remaining capacity of the storage device and allocating storage resources to the target queue according to the remaining capacity of the storage device; scheduling the storage resource location of the target queue based on the health prediction data of the storage device based on a prediction model; and storing the target queue based on the scheduled storage resource location.

[0063] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data transmission method based on dynamic routing and intelligent storage scheduling provided by the above methods. The method includes: initializing a data transmission system and receiving a message queue to be transmitted input by a user; parsing the message queue and delivering the message queue to a target queue according to a selected dynamic route; obtaining the remaining capacity of the storage device and allocating storage resources to the target queue according to the remaining capacity of the storage device; scheduling the storage resource location of the target queue based on the health prediction data of the storage device based on a prediction model; and storing the target queue based on the scheduled storage resource location.

[0065] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the data transmission method based on dynamic routing and intelligent storage scheduling provided by the above methods. The method includes: initializing a data transmission system and receiving a message queue to be transmitted input by a user; parsing the message queue and delivering the message queue to a target queue according to a selected dynamic route; obtaining the remaining capacity of a storage device and allocating storage resources to the target queue according to the remaining capacity of the storage device; scheduling the storage resource location of the target queue based on the health prediction data of the storage device based on a prediction model; and storing the target queue based on the scheduled storage resource location.

[0066] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0067] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data transmission system based on dynamic routing and intelligent storage scheduling, characterized in that, The application relates to a data transmission system, which comprises the following parts: a Kafka data input layer for receiving user inputted message queues to be transmitted; a transmission service module for parsing the message queues and delivering the message queues to target queues according to selected dynamic routing; a storage scheduling engine for allocating storage resources of the target queues according to the remaining capacity of storage devices and scheduling the storage resource positions of the target queues based on health degree prediction data of the storage devices; a physical storage pool for storing the target queues, which is composed of the storage devices.

2. The system of claim 1, wherein, The transmission service module comprises the following parts: a configuration management submodule for loading, parsing and maintaining global configuration information of a data transmission system, loading a configuration file when the data transmission system is started and initializing a global structure; a message processing submodule for parsing the message queues and extracting features of the target queues; a message routing submodule for dynamically selecting a message queue delivery path according to data content, application devices and configuration rules obtained by parsing; a message delivery submodule for delivering the parsed message queues to target queues through a data transmission bus according to the selected message queue delivery path; a log submodule for recording key events generated after message queue delivery to obtain logs, wherein the key events include delivery success, delivery failure, error recovery and configuration loading information; the log submodule classifies the logs based on the content and access frequency of the logs and distributes the classified logs to corresponding storage layers through a dynamic scheduler, wherein the content of the logs includes at least one of a timestamp, a time type and a priority; a file management submodule for managing files needed during operation of the data transmission system, wherein the management operations include configuration loading, persistent storage, log rotation and cross-platform adaptation, the persistent storage includes fault domain-aware persistent storage for executing a replica distribution strategy and cross-platform file operation, the replica distribution strategy includes a data type strategy based on configuration information, unconfirmed private messages and the number of replicas; a fault domain isolation level strategy based on a server level, a rack level and a data center level; and the cross-platform file operation includes atomic write operation support and log rotation threshold adjustment.

3. The system of claim 1, wherein, The storage scheduling engine comprises the following parts: a state perceiver for continuously monitoring middleware states and physical indexes, wherein the middleware states include the length of a transmission message queue, a task execution state and resource usage, and the physical indexes include the remaining capacity, read-write performance and failure rate of the storage devices; a strategy decision maker for making a storage strategy decision according to the information monitored by the state perceiver, a pre-set storage strategy and business demands; a prediction model for predicting the future health degree of the storage devices by using historical data and current system state information; an execution coordinator for converting the storage strategy made by the strategy decision maker into specific operation instructions and coordinating various submodules to execute corresponding operation instructions.

4. The system of claim 1, wherein, The physical storage pool is divided into a four-layer structure according to access features. The four-layer structure is sequentially arranged from top to bottom as edge storage medium, center storage cluster, near-line storage node and archive storage facility, wherein each layer of storage medium is accessed to the storage scheduling engine through a unified interface.

5. A data transmission method based on dynamic routing and intelligent storage scheduling, characterized in that, The data transmission system of any one of claims 1-4, comprising: initializing the data transmission system and receiving a message queue to be transmitted input by a user; parsing the message queue and delivering the message queue to a target queue according to a selected dynamic route; obtaining a remaining capacity of a storage device and allocating storage resources of the target queue according to the remaining capacity of the storage device; predicting a health degree of the storage device based on a prediction model and scheduling a storage resource location of the target queue based on the predicted health degree of the storage device; storing the target queue based on the scheduled storage resource location.

6. The method of claim 5, wherein, The parsing the message queue and delivering the message queue to a target queue according to a selected dynamic route, comprises: parsing the message queue to obtain data content; dynamically selecting a message queue delivery path based on the data content and an application device and a configuration rule; delivering the parsed message queue to a target queue through a data transmission bus according to the selected message queue delivery path.

7. The method of claim 5, wherein, The method further comprises: recording key events generated after message queue delivery to obtain a log, the key events including delivery success, delivery failure, error recovery and configuration loading information; classifying the log based on content and access frequency of the log and distributing the classified log to a corresponding storage layer through a dynamic scheduler, wherein the content of the log includes at least one of a timestamp, a time type and a priority.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the data transmission method based on dynamic routing and intelligent storage scheduling of any one of claims 5-7. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the data transmission method based on dynamic routing and intelligent storage scheduling of any one of claims 5-7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the data transmission method based on dynamic routing and intelligent storage scheduling of any one of claims 5-7.

Images