Data transmission method, system and device for full-link monitoring and storage medium

Abstract

The application provides a data transmission method, system and device for full-link monitoring and a storage medium, and belongs to the technical field of data processing and cross-level transmission. The system comprises a provincial system deployed at a provincial node and a departmental system deployed at a departmental platform. The provincial system extracts target data through a data acquisition module, and encapsulates the target data into a standardized data packet through a data processing and transmission module, and then transmits the data packet to the departmental system. The departmental system receives, analyzes and persistently stores the data packet through a data storage module. The method correspondingly realizes the above process. Through clear hierarchical design, the application effectively solves the problems of strong coupling, low reliability and difficult fault tracing between existing departmental and provincial systems. The architecture supports the adaptation of a domestic platform, and is convenient for integrating mechanisms such as full-link monitoring, asynchronous buffering and idempotent writing, thereby significantly improving the standardization, observability, scalability and business continuity of data transmission.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and in particular to a data transmission method, system, device, and storage medium for end-to-end monitoring. Background Technology

[0002] Building an efficient and reliable two-tier data exchange system between the ministry and provincial levels has become a critical requirement. Currently, such systems are mostly built on mainstream foreign operating systems and databases, posing risks to independent controllability and facing a series of challenges in actual operation. First, the ministry-level platform and provincial business systems typically use tightly coupled direct connections or simple file exchanges, resulting in strong inter-system dependencies. Any interface change or service fluctuation on either side can easily trigger a chain reaction of failures, leading to poor overall scalability. Second, the data transmission process lacks transparent and unified monitoring methods. Once data loss, delay, or duplication occurs, problem localization and tracing are extremely difficult, often relying on manual layer-by-layer verification, which is inefficient and fails to meet the reliability requirements of financial-grade business. Furthermore, the existing architecture is difficult to gracefully adapt to the migration trend of domestic basic software and hardware platforms, facing high implementation barriers in terms of compatibility, performance tuning, and security hardening. Therefore, there is an urgent need for a data transmission solution with a clear structure, monitorable capabilities, high reliability, and easy deployment in a domestic environment, providing end-to-end monitoring to solve the aforementioned problems of system coupling, opaque operation and maintenance, and domestic adaptation. Summary of the Invention

[0003] This invention provides a data transmission method, system, device, and storage medium for end-to-end monitoring, which solves the core problems of existing data transmission between ministries and provinces, such as system coupling, black box operation and maintenance, and poor reliability. It provides an efficient, reliable, transparent, and easy-to-deploy domestically produced overall solution.

[0004] This invention provides a data transmission system for end-to-end monitoring, comprising: The data acquisition module is used to extract target data to be transmitted from the provincial business system and push it to the message queue; The data processing and transmission module is used to consume the target data in the message queue, package the target data into a standard data packet, and then initiate transmission to the ministerial platform. The data storage module is used to persistently store status data, failure retry data, and data downloaded from the ministerial-level platform during the transmission process.

[0005] The data transmission system for end-to-end monitoring provided by the present invention also includes a transmission monitoring system deployed across two levels; The transmission monitoring system is used to embed data points in the transmission link between the provincial system and the ministerial system to generate a globally unique transmission tracking identifier for real-time monitoring of data transmission status and link backtracking.

[0006] According to the present invention, a data acquisition module for end-to-end monitoring of data transmission is provided, wherein the data acquisition module includes an Apache Kafka message queue, which is used to group, isolate, and asynchronously buffer the target data according to business type.

[0007] According to the present invention, a data transmission system for end-to-end monitoring is provided, wherein the data processing and transmission module is built based on the XXJ distributed scheduling framework and is used to schedule and consume data in the Apache Kafka message queue for encapsulation and transmission.

[0008] According to the present invention, a data transmission system for end-to-end monitoring is provided, wherein the data storage module includes a GaussDB database, which is used to receive data packets and corresponding globally unique request identifiers from provincial systems. Before performing the write operation, verify whether the request identifier already exists; If not, perform the write operation and record the request identifier; if yes, reject the duplicate write.

[0009] According to the data transmission system for end-to-end monitoring provided by the present invention, the data processing and transmission module is further configured to persistently store and automatically retry the data packet according to a preset strategy when the transmission to the departmental system fails.

[0010] According to the data transmission system with end-to-end monitoring provided by the present invention, the system is further provided with a resource monitoring module, which is used to monitor the server resources where the Apache Kafka message queue is located, and trigger an early warning or expansion operation when the disk usage exceeds a threshold.

[0011] This invention also provides a data transmission method for end-to-end monitoring, comprising: The present invention extracts target data to be transmitted from the provincial business system and pushes it to the message queue; the present invention also provides an electronic device, including a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the data transmission system with full-link monitoring as described above.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a data transmission system with end-to-end monitoring as described above.

[0013] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a data transmission system for end-to-end monitoring as described above.

[0014] The data transmission method, system, equipment, and storage medium for end-to-end monitoring provided by this invention clearly divides the system into a provincial system deployed at provincial nodes and a ministerial system deployed on a ministerial platform. Within the provincial system, it further decouples into a data acquisition module and a data processing and transmission module. This invention constructs a hierarchical architecture with single responsibilities. This design achieves effective decoupling between the ministerial and provincial platforms, as well as between data processing links within the provincial system, significantly reducing system complexity. It allows each module to be independently developed, deployed, and expanded, greatly improving the maintainability and scalability of the entire system.

[0015] Secondly, by encapsulating the raw data into standardized data packets through the data processing and transmission module, this invention establishes a unified and standardized data interaction contract between the ministerial and provincial levels. This not only simplifies the interface processing logic and improves data processing efficiency, but also lays a solid foundation for reliable data transmission and accurate parsing. At the ministerial level, the data packets are persistently stored by a dedicated data storage module, further ensuring data integrity and security.

[0016] Subsequent introduction of various highly reliable and observable enhancements provides elegant support. For example, based on this architecture, a transmission monitoring system can be naturally integrated to achieve end-to-end tracing, message queues can be used for asynchronous buffering and business isolation, a scheduling framework ensures reliable task execution, and finally, idempotent writes and other mechanisms in a Gaussian database guarantee absolute data consistency. Simultaneously, this architecture has good compatibility with domestic basic software platforms, supporting efficient system operation in an independently controllable environment. In summary, this invention fundamentally solves the core problems of system coupling, black-box operation and maintenance, and poor reliability in existing ministerial and provincial data transmission, providing a highly efficient, reliable, transparent, and easily deployable domestically. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the data transmission system for end-to-end monitoring provided by the present invention; Figure 2 This is a flowchart illustrating the data transmission method for end-to-end monitoring provided by the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] To address the problems in existing technologies, this invention proposes a full-link monitoring data transmission system. This system solves core issues in existing ministerial and provincial data transmission systems, such as system coupling, black-box operation and maintenance, and poor reliability. It provides a highly efficient, reliable, transparent, and easily deployable domestically. The following describes this full-link monitoring data transmission system. Figure 1 As shown, including but not limited to the following modules: The data acquisition module is used to extract target data to be transmitted from the provincial business system and push it to the message queue; The data processing and transmission module is used to consume the target data in the message queue, package the target data into a standard data packet, and then initiate transmission to the ministerial platform. The data storage module is used to persistently store status data, failure retry data, and data downloaded from the ministerial-level platform during the transmission process.

[0021] This embodiment provides a specific implementation of a two-tier (ministerial and provincial) data transmission system based on a domestically developed basic software platform. For example... Figure 1 As shown (corresponding to the system architecture diagram in the original disclosure document), the system is deployed at the provincial node, and its core includes a data acquisition module, a data processing and transmission module, and a data storage module. The system runs on the domestically produced Galaxy Kylin Advanced Server Operating System (V10), and achieves secure, reliable, and monitorable data exchange with the ministerial-level platform.

[0022] Detailed implementation of the data acquisition module The data acquisition module is the system's data entry point, responsible for efficiently and accurately acquiring target data to be transmitted from various provincial business systems.

[0023] Data source and extraction: The target data mainly comes from the provincial expressway network toll collection system, including but not limited to: original transaction records, gantry passage records, toll collection results, audit lists, parameter files, etc.

[0024] The data acquisition module operates through a data extraction unit deployed on the business system side. This unit supports multiple data interfaces, including: Database connection: Directly connect to the intermediary database of the business system (such as MySQL, PostgreSQL), and execute SQL statements to extract incremental or full data by scheduling on a timer or listening to database log changes (such as Debezium CDC).

[0025] File monitoring: Monitors specified file directories of business systems (such as FTP servers, shared storage), and automatically captures and parses new data files (such as CSV and JSON formats) when they are generated.

[0026] API calls: For business subsystems that provide real-time APIs, data is obtained by calling their data service interfaces.

[0027] Data is pushed to the message queue: The extracted target data is immediately pushed to a distributed message queue built on Apache Kafka.

[0028] Queue Topic Planning: To achieve data isolation and priority management, different Kafka Topics are divided according to business type, such as: topic_transaction (transaction-related), topic_toll (billing-related), topic_audit (auditing-related), and topic_param (parameter-related). Each Topic can be configured with multiple partitions based on data volume to improve parallelism.

[0029] Push Process: The data extraction unit encapsulates the data into a message body in a unified format (such as Avro or JSON), and attaches metadata such as business type, source system identifier, and extraction timestamp. Then, it calls the Kafka Producer API to send the message to the corresponding Topic. This process is asynchronous to ensure that it does not affect the performance of the source system.

[0030] Detailed Implementation of the Data Processing and Transmission Module This module is the core hub for data transmission, responsible for processing and encapsulating data in the message queue and reliably transmitting it to the departmental platform.

[0031] Data consumption and verification: Consumer service: Deploy multiple data consumption units (as Kafka consumers), subscribing to corresponding topics in the form of consumer groups. The consumption units are deployed in a cluster to achieve load balancing and high availability.

[0032] Business Validation: After retrieving messages from the queue, the consumption unit first performs strict validation according to the "Data Transmission Interface Specification" agreed upon by the Ministry and Provincial levels. Validation includes: data format compliance, field integrity, logical consistency (e.g., non-negative transaction amounts), and compliance with business rules. Messages that fail validation are marked and transferred to the "Dead-Letter Queue" for subsequent manual investigation, while an alarm is sent to the monitoring module.

[0033] Data packaging and encapsulation: Data that passes verification enters the data packaging unit. This unit is built and managed based on the domestic open-source distributed task scheduling framework XXJ (xxl-job).

[0034] Task Scheduling: One or more "data packaging tasks" are created in the XXJ framework for each business type. These tasks are configured to execute at a high frequency (e.g., once every 10 seconds) and are triggered uniformly by the XXJ scheduling center. The framework supports failover and load balancing strategies to ensure continuous task operation.

[0035] Standardized encapsulation: When the packaging task is executed, it retrieves a batch of validated data from memory or cache and encapsulates it into a standard data packet. The structure of the standard data packet includes: Message header: contains version number, unique data packet ID (UUID), business type, source province code, timestamp, total data volume, etc.

[0036] Message body: Stores the actual business data that has been serialized (such as JSON or Protocol Buffers).

[0037] Signature: The entire data packet (or message header + message body hash value) is digitally signed using the asymmetric private key of the provincial node to ensure data integrity and source credibility.

[0038] Data transmission: The encapsulated standard data packets are sent by the data transmission unit. This unit uses the HTTPS protocol to send the data packets as the request body, calling the unified data reception RESTful API provided by the ministerial platform.

[0039] Reliable transmission mechanism: Synchronization and Retry: A synchronous call method is adopted, and a reasonable timeout is set. If the first call fails (network timeout, server error, etc.), it will automatically retry according to a predefined retry strategy (such as exponential backoff).

[0040] Response processing: Receive and parse responses from the ministerial-level platform. Successful responses contain the data reception ID assigned by the ministerial-level platform. Failure responses are recorded, and the status of the corresponding data packet is set to "transmission failed".

[0041] Detailed implementation of the data storage module The data storage module provides the system with persistence capabilities, records critical states, and ensures the traceability and reliability of data transmission.

[0042] Storage Engine: The core storage engine is the domestically developed GaussDB distributed version. The database cluster is deployed with a single AZ and 3 replicas to ensure high data reliability.

[0043] Storage content and structure: This module primarily maintains the following types of key tables: Data Transmission Log Table: Records the entire lifecycle status of each data packet from acquisition to completion of transmission. Fields include: Data Packet ID, Business Type, Source Data ID, Kafka Message Offset, Verification Status, Packaging Time, Transmission Request Time, Department-Level Response Result, Department-Level Receiver ID, Final Status (Success / Failure), and Completion Time. This table is the primary basis for tracing data.

[0044] Retry Table for Transmission Failures: This table specifically stores information about data packets that failed to transmit and their original content (or pointers to object storage). It records the reason for the failure, the number of retries, and the next retry time. The data processing and transmission module periodically scans this table and re-initiates the packaging and transmission process for data that meets the retry criteria.

[0045] Departmental Download Data Table: Stores data actively downloaded or received from the departmental platform, such as the latest rate parameters, blacklists, and settlement reconciliation files. The table structure is dynamically defined based on data type and records information such as download batch, version, and effective time.

[0046] Data consistency and fault tolerance: All database write operations are completed within a transaction, ensuring strong consistency between logs and business status.

[0047] By leveraging GaussDB's idempotent write feature, when handling retry requests, we check whether the "packet ID" already exists to avoid duplicate data entry due to network jitter or other reasons.

[0048] Data and its logs that fail to be transmitted will only be archived or cleared from the "Failure Retry Table" after they are successfully transmitted to the department level and confirmed, in order to prevent data loss.

[0049] System workflow overview: Combining the modules mentioned above, a complete data upload process is as follows: The provincial toll collection system generates a new gantry transaction record, which is then written into the intermediary database.

[0050] The extraction unit of the data acquisition module 100 detects data changes, captures them, and pushes them to the Kafka topic_transaction.

[0051] The consumption unit of the data processing and transmission module 200 consumes this message from the topic and performs business verification (such as checking the vehicle license plate format and the validity of the passage time).

[0052] After verification, the XXJ framework schedules a packaging task to encapsulate the record along with several other records into a standard data packet and generate a digital signature.

[0053] The data transmission unit sends data packets to the ministerial-level platform interface via HTTPS.

[0054] At the same time, the data storage module inserts a record into the "Data Transmission Log Table" with the status "Transmitting".

[0055] The ministerial-level platform successfully received and processed the data, returning a success response and the receiving ID.

[0056] The data transmission unit updates the log table, changes the status to "success", and records the department-level receiving ID.

[0057] If the transmission fails, the log table status is changed to "failed", and the core information of the data packet is stored in the "transmission failure retry table" for subsequent retries.

[0058] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0059] As a further optional embodiment, it also includes a transmission monitoring system deployed across two levels; The transmission monitoring system is used to embed data points in the transmission link between the provincial system and the ministerial system to generate a globally unique transmission tracking identifier for real-time monitoring of data transmission status and link backtracking.

[0060] As a further optional embodiment of the present invention, the end-to-end monitoring data transmission system also includes a transmission monitoring system deployed across two levels. This system is independent of the core data transmission process and focuses on providing observability assurance for the entire link.

[0061] This transmission monitoring system typically consists of a centralized monitoring platform and distributed probes deployed in key components of provincial and ministerial-level systems. Its core working principle is as follows: Generation and transmission of trace identifiers: When the data acquisition module of the provincial system extracts a piece of target data from the business system, it immediately generates a globally unique transmission trace identifier (Trace ID), such as a UUID. This identifier serves as the "identity card" for the data and is forcibly transmitted in all subsequent stages.

[0062] End-to-end tracking: The system tracks data at all critical nodes along the data transmission path. In the provincial system: when data is placed in the message queue, consumed by the data processing and transmission module, encapsulated, and a network request is sent to the ministerial level, the probes of the relevant components will record event logs with the current Trace ID and timestamp.

[0063] At the ministerial level: When the data receiving service receives a request, completes verification, begins writing to the database, and ultimately writes successfully or unsuccessfully, the corresponding events are also recorded.

[0064] All logs are sent asynchronously to the central monitoring platform.

[0065] Monitoring, backtracking, and alerting: After collecting all logs, the monitoring platform connects events scattered across multiple nodes at two levels using Trace IDs, reconstructing the complete transmission path and lifecycle of a single data entry. Operations personnel can view the link health status, throughput, and latency in real time through the "Data Map" view on the platform. When any link experiences a timeout or error, the platform can quickly locate the fault point and accurately backtrack the complete flow of problematic data using the associated Trace ID, greatly improving troubleshooting efficiency. Simultaneously, the system can configure real-time alerts for abnormal states such as consecutive failures or link interruptions.

[0066] This transmission monitoring system uses technical means to achieve "white-box" management of complex cross-level data transmission processes, transforming invisible network communication and data flow into visible, measurable, and traceable standardized operation and maintenance objects. It is a key facility to ensure the high reliability and maintainability of the system.

[0067] As a further optional embodiment, the data acquisition module includes an Apache Kafka message queue for grouping, isolating, and asynchronously buffering the target data according to business type.

[0068] As a further optional embodiment of the present invention, the data acquisition module of the provincial system integrates an Apache Kafka message queue, whose core function is to group and isolate target data and asynchronously buffer it in order to optimize the reliability and scalability of the system architecture.

[0069] Specifically, after the data acquisition module successfully extracts the target data from the provincial business system, it is not directly handed over to subsequent modules for processing. Instead, based on the data's business attributes (e.g., toll transactions, vehicle credit lists, evidence of overloading, audit work orders, etc.), it is published to pre-created corresponding topics in Kafka. For example, all transaction data is published to topic_transaction, and all list data is published to topic_blacklist. This design achieves physical isolation at the business level, ensuring that data streams from different businesses do not interfere with each other, and that backlog or processing anomalies in any business queue will not affect the normal transmission of other businesses.

[0070] Meanwhile, Kafka, as a high-throughput distributed messaging system, acts as an efficient asynchronous buffer here. Once the data acquisition module writes data to Kafka, it considers its current task complete and can immediately continue to the next round of extraction without waiting for the data to be actually processed and uploaded. The speed of data consumption and processing is independently controlled by subsequent data processing and transmission modules based on their own capabilities and the load of the departmental interface. This decoupling mechanism between production and consumption effectively smooths out peak and valley loads, effectively handling instantaneous data surges generated by provincial business systems, avoiding impacts on downstream processing modules and departmental interfaces, thereby significantly improving the stability and throughput of the entire transmission link.

[0071] As a further optional embodiment, the data processing and transmission module is built on the XXJ distributed scheduling framework and is used to schedule and consume data in the Apache Kafka message queue for encapsulation and transmission.

[0072] As a further optional embodiment of the present invention, the data processing and transmission module of the provincial system is built on the XXJ distributed scheduling framework (e.g., XXL-JOB), and its core function is to realize task-oriented scheduling, reliable consumption and standardized encapsulation of data processing.

[0073] Specifically, the data processing and transmission module is typically divided into two parts in its architecture: The dispatch center, acting as the control hub, is a separately deployed service used for unified management, monitoring, and triggering of all data upload tasks. Administrators can configure independent scheduling tasks for different business themes (such as transactions and lists) on its web console, setting execution cycles (e.g., once every 30 seconds), routing strategies, and fault tolerance mechanisms.

[0074] Executor Cluster: As task execution units, they are deployed in a cluster within the provincial system. Each executor is an application that embeds specific business logic.

[0075] Its workflow is as follows: When the scheduling center triggers a "transaction data upload task," it distributes the task to an available executor according to a preset routing strategy (such as round-robin). Upon receiving the scheduling request, the executor immediately starts its internal processing logic: a. Targeted consumption: The executor acts as a consumer, connecting to Apache Kafka and pulling a batch of messages to be processed from a specified topic (e.g., topic_transaction).

[0076] b. Business encapsulation: The processing program in the executor encapsulates the raw business data it pulls into a standard data packet in a unified format, strictly in accordance with the interface protocol (including message structure, field mapping, data signature algorithm, etc.) specified by the department-level platform.

[0077] c. Reliable Transmission and Status Feedback: The executor sends data packets to the receiving interface of the departmental system via an HTTP / HTTPS client. It handles network interactions, manages connection timeouts and retries. Regardless of success or failure, the executor sends a log of the task execution results (including the amount of data processed, time taken, and final status) back to the scheduling center, forming a complete scheduling log for auditing.

[0078] By introducing the XXJ scheduling framework, this embodiment upgrades the data upload action from a simple code call to a standardized task with visual configuration, distributed execution, and end-to-end monitoring. It enables precise control over the data processing rhythm and ensures that tasks can be automatically retried or alerted when anomalies occur during Kafka consumption, encapsulation, or transmission, thereby greatly enhancing the controllability and reliability of the transmission process.

[0079] As a further optional embodiment, the data storage module includes a GaussDB database, which is used to receive data packets from the provincial system and the corresponding globally unique request identifier. Before performing the write operation, verify whether the request identifier already exists; If not, perform the write operation and record the request identifier; if yes, reject the duplicate write.

[0080] As a further optional embodiment of the present invention, the core of the data storage module of the departmental system adopts GaussDB as the persistent storage engine. Its key innovation lies in the implementation of an idempotent write mechanism based on a globally unique request identifier, so as to completely solve the data duplication problem that may be caused by network retransmission, system retries, etc.

[0081] In practice, the data processing and transmission module of the provincial system generates a globally unique request ID (such as a UUID string) for each standard data packet when encapsulating it, and places this ID in the packet header. After the data packet is transmitted to the ministerial system, the data storage module's data receiving and writing services execute the following atomic operation process: Reception and Extraction: After receiving the data packet, the service extracts the globally unique request identifier from the packet header while parsing the business data.

[0082] Atomicity check: Before actually executing the database INSERT operation, the service first queries a special "request record table" in GaussDB or uses the database's unique index feature to perform an existence check, using the request identifier as a condition. This "query-determine" logic and the subsequent write operation must be encapsulated in a database transaction to ensure its atomicity.

[0083] Conditional write: If the verification passes (identifier does not exist): This indicates that the request is arriving for the first time. The system then performs two operations: a) writes the core business data into the corresponding business fact table; b) records the globally unique request identifier into the "Request Record Table". Both operations are completed within the same transaction, ensuring that they either succeed or fail simultaneously, and then a success response is returned to the provincial system.

[0084] If the validation fails (indicating the request already exists): this means the request has already been processed, and the current data packet is a duplicate request. The system will immediately abandon the current data write operation and return a "operation successful" response to the provincial system, thus ensuring the idempotency of the business logic.

[0085] This embodiment intercepts duplicate data at the entry point through lightweight logic checks at the database front end, avoiding complex deduplication queries at the business table level and greatly improving write efficiency and data consistency. It allows the entire transmission system to confidently use retry strategies to address network instability without worrying about data redundancy at the departmental level, making it a key technological foundation for building a highly reliable data aggregation platform.

[0086] As a further optional embodiment, the data processing and transmission module is also used to persistently store and automatically retry the data packet according to a preset strategy when the transmission to the departmental system fails.

[0087] As a further optional embodiment of the present invention, the data processing and transmission module of the provincial system integrates a highly reliable failure retry and persistent storage mechanism. This mechanism aims to ensure that even in abnormal situations such as temporary network interruptions or temporary unavailability of departmental services, data packets will not be lost and can be automatically and reliably retransmitted after conditions are restored, thereby achieving extremely high transmission reliability.

[0088] Specifically, after the data processing and transmission module (through its XXJ task executor) sends a data packet to the ministerial-level system interface via an HTTP client, the task is not immediately considered complete. Its subsequent processing logic is as follows: Failure Assessment and Persistence: If the HTTP client returns a network error (such as connection timeout or connection refused) or receives an explicit failure status code (5xx) from the server, the transmission is considered a "failure." The executor does not discard this data packet but immediately persists it (including the complete standard data packet content and the associated globally unique request identifier) ​​to a highly available local storage medium. This medium is typically a separate relational database table (such as in MySQL) or a local file storage system, recording the failure time, reason for failure, and the current number of retries (initially 0).

[0089] Policy-based automatic retries: The system has pre-configured flexible retry policies. For example, it can be configured as "exponential backoff retries": after the first failure, wait 1 minute to retry; after the second failure, wait 2 minutes; after the third failure, wait 4 minutes, and so on, until the maximum number of retries (e.g., 5 times) is reached. The retry task can be triggered by a dedicated "failure retry task" created by the XXJ scheduling framework to periodically scan the failure records in persistent storage, or the original task can trigger the retry logic itself in subsequent scheduling cycles.

[0090] State Management and Final Processing: Each retry updates the retry count and status of the data packet record. If a retry succeeds, the record status is marked as "Successful" and can be archived or deleted from persistent storage. If the maximum number of retries is reached and the packet still fails, the record status is marked as "Final Failure." For data packets that "Final Failure," the system triggers advanced alerts (such as sending SMS or email notifications to operations personnel) and may move them to a dedicated "dead letter" queue or table for manual intervention and possible subsequent remedial processing.

[0091] This embodiment manages transient failures in memory by converting them into a persistent state, enabling data transmission to have the capabilities of "resuming interrupted transmissions" and "self-recovering from failures." It effectively addresses unavoidable transient failures in production environments and is a key technical component for ensuring data is "not lost or duplicated" and achieving the promise of zero system service interruption.

[0092] As a further optional embodiment, the system also includes a resource monitoring module for monitoring the server resources where the Apache Kafka message queue is located, and triggering an alert or expansion operation when the disk usage exceeds a threshold.

[0093] As a further optional embodiment of the present invention, the system also includes an independent resource monitoring module, which is specifically used to monitor and protect the server hosting the Apache Kafka message queue cluster that supports core data transmission at the infrastructure level.

[0094] This module is typically deployed as a standalone proxy service or monitoring probe on each node of the Kafka cluster to continuously collect key system resource metrics, with disk usage being the core monitoring target. Because all Kafka messages are persistently stored on disk, running out of disk space will directly lead to queue blocking, data write failures, and ultimately, an interruption of the entire data transmission process.

[0095] Its workflow is as follows: The resource monitoring module periodically (e.g., every minute) checks the usage of the disk partition where the Kafka data log directory is located. The system presets one or more thresholds, for example: Warning threshold: When disk utilization reaches 80%, the module will immediately trigger a warning. The warning information will be notified to system maintenance personnel through integrated alarm channels (such as email, SMS, or internal office software bots), indicating "Kafka cluster disk space is low," and attaching specific node, directory, and utilization information for manual intervention and handling.

[0096] Automatic scaling / cleanup thresholds: When disk utilization reaches a more urgent level of 90% or 95%, in addition to sending higher-level alerts, the module can also trigger proactive intervention operations based on preset automation policies. In cloud environments or containerized deployments, this might mean automatically performing storage scaling to dynamically increase disk space for nodes. In physical environments, the policy might be linked to log cleanup tasks, automatically cleaning up old data files in Kafka that have exceeded their retention period (such as logs older than 7 days), or stopping data writes to the slowest topic to quickly free up space and prevent the service from becoming completely unavailable.

[0097] This embodiment extends system stability assurance from the application logic layer to the infrastructure layer. Through proactive resource monitoring and automated intervention of Kafka, a critical dependency component, it effectively prevents global transmission failures caused by the exhaustion of underlying resources, making it an important operational support component for achieving high availability and continuous service capabilities.

[0098] The system of this invention can be deployed on a domestically developed basic software platform. In this embodiment, the server nodes of the provincial and ministerial-level systems can be preferentially deployed on a domestically developed secure operating environment, represented by the Galaxy Kylin Advanced Server Operating System.

[0099] 1. System Deployment Environment The provincial-level systems deployed at various provincial nodes, including their data acquisition, processing, and transmission modules, as well as the data storage module and database deployed on the ministerial-level central platform, can all be installed and run on compatible domestic operating systems. For example, core middleware and databases such as Apache Kafka message queues, the XXJ distributed scheduling framework, and GaussDB have all been adapted to mainstream domestic operating systems and can run stably and efficiently in this environment.

[0100] 2. Common support provided by domestically developed operating systems When the system is deployed on such a domestically developed operating system, its system-level capabilities can be fully utilized to enhance the overall reliability and security of the solution: Security Enhancement Support: By leveraging the security enhancement mechanisms embedded in the operating system, mandatory access control policies can be configured for critical data processing processes to strictly restrict their system resource access permissions and achieve effective security isolation.

[0101] Unified service governance: Application services can be managed in a unified lifecycle through standard operating system service management tools (such as systemd), enabling highly reliable service protection and self-recovery from failures, and facilitating centralized collection of system logs.

[0102] Fine-grained access control: Based on the operating system's comprehensive access management model, role-based access control can be implemented to ensure that all operation and maintenance permissions are clear and auditable, meeting high standards of security management requirements.

[0103] Through the above implementation methods, this invention demonstrates the complete deployment and operation capabilities of the system on a domestically developed technology stack. This solution not only achieves core functions but also, by interfacing with the underlying security and management features of domestic operating systems, enables the entire end-to-end monitoring data transmission system to achieve enterprise-level high reliability, high security, and ease of maintenance within an independently controllable hardware and software environment.

[0104] The data transmission method for end-to-end monitoring provided by this invention is described below, such as... Figure 2 As shown, the data transmission method for end-to-end monitoring described below can be referred to in correspondence with the data transmission system for end-to-end monitoring described above.

[0105] A data transmission method for end-to-end monitoring, comprising: Step 210: Extract the target data to be transmitted from the provincial business system through the provincial system deployed at the provincial node; Step 220: Encapsulate the target data into standardized data packets through the provincial system; Step 230: Transmit the data packet to the ministerial-level system deployed on the ministerial-level platform through the provincial-level system; Step 240: Receive, parse, and persistently store data packets from the provincial system through the ministerial-level system.

[0106] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a data transmission system with end-to-end monitoring. This method includes: The system extracts target data to be transmitted from the provincial business system and pushes it to the message queue. Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the end-to-end monitoring data transmission system provided by the methods described above, the method comprising: The invention extracts target data to be transmitted from the provincial business system and pushes it to a message queue. Furthermore, the invention provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the data transmission system with end-to-end monitoring provided by the methods described above. This method includes: The target data to be transmitted is extracted from the provincial business system and pushed to the message queue. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data transmission system for end-to-end monitoring, characterized in that, Deployed at provincial nodes for data exchange with ministerial-level platforms, the system includes: The data acquisition module is used to extract target data to be transmitted from the provincial business system and push it to the message queue; The data processing and transmission module is used to consume the target data in the message queue, package the target data into a standard data packet, and then initiate transmission to the ministerial platform. The data storage module is used to persistently store status data, failure retry data, and data downloaded from the ministerial-level platform during the transmission process.

2. The data transmission system for end-to-end monitoring according to claim 1, characterized in that, It also includes a transmission monitoring module; The transmission monitoring module is used to embed data points in the transmission link between provincial and ministerial nodes to generate globally unique transmission tracking identifiers for real-time monitoring of data transmission status and link backtracking.

3. The data transmission system for end-to-end monitoring according to claim 1, characterized in that, The data acquisition module includes an Apache Kafka message queue, which is used to group, isolate, and asynchronously buffer the target data according to business type.

4. The data transmission system for end-to-end monitoring according to claim 1, characterized in that, The data processing and transmission module is built on the XXJ distributed scheduling framework and is used to schedule and consume data in the Apache Kafka message queue for encapsulation and transmission.

5. The data transmission system for end-to-end monitoring according to claim 1, characterized in that, The data storage module includes a GaussDB database, which is used to receive data packets from the provincial system and the corresponding globally unique request identifier. Before performing the write operation, verify whether the request identifier already exists; If not, perform the write operation and record the request identifier; if yes, reject the duplicate write.

6. The data transmission system for end-to-end monitoring according to claim 1, characterized in that, The data processing and transmission module is also used to persistently store and automatically retry the data packets according to a preset strategy when the transmission to the ministerial system fails.

7. The data transmission system for end-to-end monitoring according to claim 1, characterized in that, The system also includes a resource monitoring module, which monitors the server resources where the Apache Kafka message queue is located, and triggers an alert or expansion operation when the disk usage exceeds a threshold.

8. A data transmission method for end-to-end monitoring, characterized in that, The method includes: Extract the target data to be transmitted from the provincial business system and push it to the message queue; Consume the target data in the message queue, package the target data into a standard data packet, and then initiate transmission to the ministerial-level platform; Persistent storage of state data during transmission, failure retry data, and data downloaded from the ministerial-level platform.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data transmission method for end-to-end monitoring as described in any one of claims 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data transmission method for end-to-end monitoring as described in any one of claims 8.

Images