Cross-network data transmission monitoring implementation method based on full link tracking

Abstract

The invention discloses a cross-network data transmission monitoring implementation method and system based on full-link tracking, and belongs to the technical field of data transmission monitoring. The method comprises the following steps: defining a plurality of key monitoring nodes on a data transmission path crossing a first network and a second network; collecting the state and timestamp data of each node by adopting a non-intrusive technology; a unified transaction ID is generated for each transmission task, the monitoring data belonging to different networks are stored in an associated manner by using the ID, and an end-to-end full-link tracking view is constructed; performing real-time analysis and early warning on the transmission process based on a preset rule; and finally, displaying the transmission state in a unified visual interface. According to the invention, the problems of large monitoring blind area, difficulty in problem positioning and early warning lagging of cross-network data transmission in the prior art are solved, and the whole data transmission process is controllable, checkable and early-warning.

Description

Technical Field

[0001] This invention relates to the field of data transmission monitoring technology, and specifically to a method for monitoring cross-network data transmission based on end-to-end tracing. Background Technology

[0002] Currently, in complex information systems such as supply chain management, there are often multiple logically or physically isolated networks (e.g., internal and external networks), and data interaction between them is crucial for business collaboration. Cross-network data transmission (the process of exchanging and transmitting data between two or more logically or physically isolated networks) serves as a bridge connecting different networks, and its stability, security, and traceability are of paramount importance.

[0003] In existing technologies, monitoring of data transmission is mostly limited to a single network, such as through application log analysis, network traffic monitoring, or single-point monitoring scripts.

[0004] The above-mentioned methods for monitoring cross-network data transmission have the following main drawbacks: Lack of end-to-end perspective: The above technologies all focus on a single network or a single system and cannot connect multiple operation nodes across two independent networks to form a complete monitoring view; that is, they cannot monitor the complete path from data initiation in network A to consumption in network B, and the cross-network link becomes a blind spot.

[0005] When transmission is abnormal, because the logs of different networks are independent and lack correlation, it is difficult for maintenance personnel to quickly locate the faulty node; it requires manual comparison of timestamps and content between multiple systems, which is extremely inefficient and prone to errors.

[0006] Weak early warning capabilities: It mainly relies on monitoring system infrastructure resources (such as CPU and memory) or network connectivity, and cannot set effective proactive early warning rules based on cross-network transmission business logic (such as files must be obtained by the other end within a specified time).

[0007] There is a certain degree of intrusion: Some solutions require business systems to modify their code to output monitoring logs, which is intrusive to the existing system and increases system complexity and maintenance costs.

[0008] Therefore, there is an urgent need for a solution that can achieve full-link monitoring of cross-network data, accurately locate problems, and provide timely early warnings. Summary of the Invention

[0009] To address this, the present invention provides a method for monitoring cross-network data transmission based on end-to-end tracing, thereby solving the problems of large blind spots, difficulty in problem location, and delayed early warning in existing cross-network data transmission monitoring technologies.

[0010] To achieve the above objectives, the present invention provides the following technical solution: A method for monitoring cross-network data transmission based on end-to-end tracing includes the following steps: S1: Define multiple key monitoring nodes on the data transmission path spanning the first network and the second network; S2: Collect monitoring data from the key monitoring nodes, wherein the monitoring data includes at least an operation timestamp and status information; S3: Generate a globally unique unified transaction ID for each cross-network data transmission task, and use this unified transaction ID to associate and store monitoring data belonging to the first network and the second network respectively; S4: Perform real-time analysis based on the associated end-to-end monitoring data and trigger early warnings according to preset rules; S5: Provides a visual interface to display a full-link tracing view of cross-network data transmission generated based on the unified transaction ID.

[0011] Furthermore: the key monitoring nodes include: The data submission node (node ​​1) located within the first network is used to monitor the operation of business services transferring data to cross-network data exchange services; The file generation node (node ​​2) located within the first network is used to monitor the operation of cross-network data exchange service writing data to the file public area; The file retrieval node (node ​​3) located in the second network is used to monitor the operation of cross-network data exchange service to retrieve files from the file public area; The data consumption node (node ​​4) located in the second network is used to monitor the operation of business services obtaining data from the exchange service.

[0012] Furthermore: In step S2, the method for collecting monitoring data includes at least one of non-intrusive aspect-oriented programming, file system event listening, and scheduled task probe; The monitoring data of the data submission node (node ​​1) and the data consumption node (node ​​4) are collected and implemented using non-intrusive aspect-oriented programming (AOP) technology. The monitoring data of the file generation node (node ​​2) is collected by listening to file system events. The monitoring content includes at least the file write time and file integrity verification information. The monitoring data of the file acquisition node (node ​​3) is collected by periodically scanning the public area of ​​the file by deploying a timed task probe, and the file acquisition time is recorded.

[0013] Furthermore: In step S4, the preset rules include: Set a time threshold for the file acquisition node. When it is detected that a file has been generated by the file generation node but has not been acquired by the file acquisition node within the specified time threshold, an early warning message is automatically triggered. The status of the file generation node is monitored, and an early warning message is automatically triggered when a file write failure is detected.

[0014] Furthermore, in step S2, the monitoring of the file generation node includes file integrity verification information.

[0015] Furthermore, in step S3, the monitoring data of the second network is periodically synchronized to the unified database of the first network for associated storage and querying.

[0016] Furthermore, the full-link tracing view displays transmission tasks in a list format and supports viewing the link details of any task by clicking; the link details integrate and display the monitoring data of all relevant nodes belonging to the first network and the second network.

[0017] Furthermore, the troubleshooting process for abnormal file generation nodes is as follows: 1) Service load check: First, system or operations personnel need to check the operational status of the cross-network data exchange service responsible for file generation; with a focus on evaluating its resource utilization metrics. If the check finds that the CPU or memory utilization exceeds the preset threshold, it is determined to be a resource bottleneck, and service expansion should be performed immediately to improve service processing capacity. If the resource utilization rate is normal, proceed to the next step of the investigation; 2) Storage I / O performance check: After confirming that the computing resources are normal, the input / output performance of the underlying storage system needs to be diagnosed. Use professional disk performance testing tools to perform the test; if the test results show that the IOPS is lower than the established standard, it indicates that there is a bottleneck in the storage write speed, and further investigation is needed; if the I / O performance test results are normal, then look at other possibilities. 3) Hard drive status and file lock check: Check the hard drive status to confirm whether there are physical bad sectors or logical errors on the storage hard drive, and ensure that the hardware is in good condition; check the file locks, as file lock conflicts are very likely to occur in scenarios such as concurrent writing; in addition, it is also necessary to check whether the target file is abnormally locked. If a file lock conflict exists, a strategy of forcibly releasing the file lock and triggering a retry mechanism is adopted to attempt to resume the file writing process; if there is no file lock exception, it indicates that the problem is more hidden. 4) Log analysis: When no obvious anomalies are found at the hardware and resource levels, it is necessary to conduct in-depth analysis of the detailed logs of cross-network data exchange services and application systems to find key information and locate the root cause.

[0018] Furthermore, the process for troubleshooting and handling timeouts at the file retrieval node is as follows: 1) Triggering alarms and manual intervention: The monitoring system will first automatically generate a file acquisition node timeout alarm and notify the operation and maintenance personnel to manually intervene through a predetermined channel; 2) Check the service status of the exchange platform: After receiving the alarm, the operation and maintenance personnel check the status of the cross-network data exchange service on the external network side to confirm whether it is running normally and whether there is a process freeze or service crash. 3) Verify network connectivity: If the service is in good condition, the network connectivity between the external network and the public file area needs to be verified to rule out network-level isolation or interruption issues. Connectivity verification includes checking whether the network link is working properly, whether DNS resolution is normal, and whether the firewall policy allows access. 4) Check file directory permissions: If network connectivity is also normal, then you need to check the access permissions of the external network exchange service account to the target file or its directory in the file public area.

[0019] The present invention has the following advantages: The present invention realizes true end-to-end panoramic monitoring of cross-network data transmission, making the whole process transparent and completely eliminating monitoring blind spots in cross-network links; through full-link tracing, the problem can be quickly delineated to a specific node in a specific network, which greatly shortens the troubleshooting time and improves operation and maintenance efficiency. A complete data chain is built by using a unified transaction ID, which not only facilitates troubleshooting but also provides a solid data foundation for data transmission auditing, performance analysis, and business decision-making.

[0020] Other features and advantages of the present invention will be set forth in the following description. Attached Figure Description

[0021] To more intuitively illustrate the prior art and this application, exemplary drawings are provided below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary drawings, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).

[0022] Figure 1This is a schematic diagram of the cross-network data transmission architecture and monitoring nodes for a cross-network data transmission monitoring implementation method based on full-link tracing, provided in an embodiment of this application.

[0023] Figure 2 This is a diagram of the overall monitoring system architecture according to an embodiment of the present invention.

[0024] Figure 3 This is a flowchart illustrating the workflow of the internal network monitoring module in this embodiment of the invention.

[0025] Figure 4 This is a flowchart illustrating the workflow of the external network monitoring module in this embodiment of the invention.

[0026] Figure 5 This is a flowchart of the anomaly investigation and handling process for node 2 in an embodiment of the present invention.

[0027] Figure 6 This is a flowchart of the timeout troubleshooting process for node 3 in an embodiment of the present invention. Detailed Implementation

[0028] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these embodiments are merely for further explanation of the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention based on the above-described content.

[0029] This embodiment takes the data exchange between the "internal network" and the "external network" in the supply chain system as the application scenario; it provides a method for monitoring cross-network data transmission based on full-link tracing, the core idea of ​​which is "layered collection, cross-network association, unified analysis, and centralized display".

[0030] Please see Figures 1-6 A method for monitoring cross-network data transmission based on end-to-end tracing includes the following steps: (1) Construct a monitoring system architecture that includes a data acquisition layer, a data processing layer, a monitoring and analysis layer, and a visualization layer; The data acquisition layer involves deploying monitoring domains on both the internal and external networks. The internal network monitoring domain uses Spring AOP to implement non-intrusive data entry for node 1; file generation events of node 2 are monitored through file system listening technology. The external network monitoring domain uses Quartz scheduled tasks as probes for node 3 to scan files and obtain status; it also uses aspect-oriented programming to capture business consumption events of node 4. The data processing layer can clean the data, and the cleaned data is then stored and linked in a unified manner. Data cleaning includes unifying timestamps and field formats; unified storage and association involve generating a unified transaction ID for each transmission task, using this ID to associate and store cross-network logs, and building end-to-end tracking capabilities. The monitoring and analysis layer can scan the processed log data in real time and identify abnormal events based on rules; The visualization layer provides a unified monitoring page on the internal network side, displaying end-to-end data in list and detail formats, and supporting filtering and searching.

[0031] (2) Define key monitoring nodes In this embodiment, four key monitoring nodes and their monitoring content are clearly defined; specifically: Node 1: Internal network service transmits data to cross-network data exchange service.

[0032] Node 2: The internal network cross-network data exchange service writes data to the file public area.

[0033] Node 3: External network cross-network data exchange service obtains files from the file public area.

[0034] Node 4: External network service obtains data from the switching service.

[0035] (3) Collect monitoring data from the internal network monitoring domain and the external network monitoring domain; The internal network monitoring domain collects monitoring data from nodes 1 and 2. At node 1, Spring AOP technology is used to perform non-intrusive instrumentation on business services, automatically capturing information such as request arrival time and data format. At node 2, a file system monitoring service is deployed. When a file is written to the file public area, the file write time, file size and MD5 value are recorded in real time.

[0036] The monitoring data of Node 1 and Node 2, along with a unified transaction ID, are stored in the internal network monitoring database.

[0037] The external network monitoring domain collects monitoring data from nodes 3 and 4. At node 3, a Quartz-based scheduled task is deployed to periodically scan the common area of ​​files and record the time when files are successfully acquired. If the difference between the file generation time and the current time exceeds the configured threshold, an alert is triggered. At node 4, AOP aspects are used to capture consumption events of business services and record data arrival times.

[0038] The monitoring data of nodes 3 and 4, along with the same unified transaction ID, are stored in the external network monitoring database and periodically synchronized to the internal network unified database.

[0039] (4) Data processing and display On the internal network side, the monitoring system uses a unified transaction ID from a unified database to link monitoring data from the two network domains, forming a complete tracking record.

[0040] Develop a monitoring page that displays all transmission tasks in a list format; users can click on any task to view its full-link details, including the precise time, status, and possible abnormal alarms for each node (such as node 2 failure highlighted in red, node 3 timeout notification, etc.).

[0041] (5) Establish a rule-based early warning mechanism; The early warning mechanism includes status marking and interface alarms for file writing failures on node 2; and setting time thresholds for file acquisition on node 3, which will automatically trigger an early warning task if the timeout is exceeded.

[0042] When the system triggers an alert (such as node 3 timeout), the monitoring page will highlight it and notify the operations and maintenance personnel. The operations and maintenance personnel can then follow the flowchart to check the transmission service status, network connectivity, etc., until the problem is resolved.

[0043] (6) Provide a visual interface to display a cross-network data transmission full-link tracking view generated based on the unified transaction ID.

[0044] See Figure 5 The troubleshooting process for node 2 (file generation node) is shown below. When the monitoring system detects or issues a warning that a file has failed to be written to the file's public area (i.e., node 2) via the internal network cross-network data exchange service, the following systematic and hierarchical troubleshooting and processing process will be triggered: 1) Service load check: First, system or operations personnel need to check the operational status of the cross-network data exchange service responsible for generating the files; with a focus on evaluating its resource utilization indicators.

[0045] If the check finds that the CPU or memory utilization exceeds the preset threshold, it is determined to be a resource bottleneck, and service expansion should be performed immediately to improve service processing capacity.

[0046] If the resource utilization rate is normal, proceed to the next step of the investigation.

[0047] 2) Storage I / O performance check: After confirming that the computing resources are normal, the input / output performance of the underlying storage system needs to be diagnosed. Use professional disk performance testing tools (such as iometer or FIO) to test; if the test results show that IOPS (input / output operations per second) is lower than the established standard, it indicates that there is a bottleneck in the storage write speed and further investigation is needed; if the I / O performance test results are normal, then look at other possibilities.

[0048] 3) Hard drive status and file lock check: Check the hard drive status to confirm whether there are physical bad sectors or logical errors, ensuring the basic health of the hardware; check file locks, as file lock conflicts are very likely to occur in scenarios such as concurrent writing; in addition, it is also necessary to check whether the target file is abnormally locked.

[0049] If a file lock conflict exists, a strategy of forcibly releasing the file lock and triggering a retry mechanism is adopted to attempt to resume the file writing process; if there is no file lock exception, it indicates that the problem may be more hidden.

[0050] 4) Log analysis: If no obvious anomalies are found at the hardware and resource levels, the root cause of the problem is likely to lie at the application logic or system interaction level. In this case, it is necessary to conduct in-depth analysis of the detailed logs of the cross-network data exchange service and application system to find key information such as error stacks, exceptions thrown, or permission verification failures, so as to locate the root cause.

[0051] See Figure 6 The timeout troubleshooting process for node 3 (file retrieval node) is shown below. When the monitoring system determines, based on a preset time threshold, that the external network cross-network data exchange service has timed out while retrieving a file from the public file area (i.e., node 3) and triggers an alert, the processing flow is as follows: 1) Triggering alarms and manual intervention: The monitoring system will first automatically generate a timeout alarm for node 3, and then notify the maintenance personnel to manually intervene through predetermined channels (such as SMS, email, and red highlighting on the monitoring screen); ensuring that the anomaly can be detected in a timely manner.

[0052] 2) Check the service status of the exchange platform: After receiving the alarm, the operation and maintenance personnel first check the status of the cross-network data exchange service on the external network side to confirm whether it is running normally and whether there is a process freeze or service crash. 3) Verify network connectivity: If the service is in good condition, the network connectivity between the external network and the public file area needs to be verified. The connectivity verification includes checking whether the network link is unobstructed, whether DNS resolution is normal, and whether the firewall policy allows access, in order to rule out network-level isolation or interruption issues. 6) Check file directory permissions: If network connectivity is also normal, then you need to check the access permissions of the external network exchange service account to the target file or its directory in the file public area; insufficient permissions are a common reason why files cannot be successfully retrieved.

[0053] Through the above implementation methods, the present invention achieves effective monitoring and management of the entire process of cross-network data transmission.

[0054] This invention enables true end-to-end panoramic monitoring of cross-network data transmission, making the entire process transparent and completely eliminating monitoring blind spots in cross-network links. Through full-link tracing, problems can be quickly pinpointed to specific nodes in specific networks, greatly shortening troubleshooting time and improving operation and maintenance efficiency.

[0055] This invention establishes a proactive early warning mechanism based on business logic, which can detect potential risks before they affect business operations, transforming passive response into proactive intervention and improving system stability. By constructing a complete data chain through a unified transaction ID, it not only facilitates problem investigation but also provides a solid data foundation for data transmission auditing, performance analysis, and business decision-making. It adopts a non-intrusive or low-intrusive data collection method, which has minimal impact on existing business systems, has a low technical implementation threshold, and is easy to modify and deploy on the basis of existing systems.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for implementing cross-network data transmission monitoring based on full-link tracking, characterized in that, The method comprises the following steps: S1: defining a plurality of key monitoring nodes on a data transmission path across a first network and a second network; S2: collecting monitoring data of the key monitoring nodes, the monitoring data comprising at least operation timestamps and state information; S3: generating a globally unique uniform transaction ID for each cross-network data transmission task, and using the uniform transaction ID to associate and store the monitoring data of the first network and the second network; S4: performing real-time analysis based on the associated full-link monitoring data, and triggering an early warning according to a preset rule; S5: providing a visual interface to display a cross-network data transmission full-link tracking view generated based on the uniform transaction ID. 2.The method of claim 1, wherein, The key monitoring nodes comprise: a data submission node located in the first network, configured to monitor the operation of a business service delivering data to a cross-network data exchange service; a file generation node located in the first network, configured to monitor the operation of the cross-network data exchange service writing data into a file public area; a file acquisition node located in the second network, configured to monitor the operation of the cross-network data exchange service acquiring a file from the file public area; a data consumption node located in the second network, configured to monitor the operation of the business service acquiring data from the exchange service. 3.The method of claim 1, wherein, In step S2, the method of collecting monitoring data comprises at least one of non-intrusive aspect-oriented programming, file system event listening, and a timing task probe.

4. The method of claim 2, wherein the method further comprises: In step S4, the preset rule comprises: setting a time threshold for the file acquisition node, and automatically triggering an early warning information when it is monitored that a file generated at the file generation node is not acquired by the file acquisition node within the time threshold; monitoring the state of the file generation node, and automatically triggering an early warning information when it is monitored that file writing fails.

5. The method of claim 4, wherein, In step S2, the monitoring of the file generation node comprises file integrity check information.

6. The method of claim 1, wherein the method further comprises: In step S3, the monitoring data of the second network is regularly synchronized to the unified database of the first network for associated storage and query.

7. The method of claim 1, wherein, The full-link tracking view displays transmission tasks in a list form, and supports viewing link details of any task by clicking; The link details integrate and display the monitoring data of all related nodes belonging to the first network and the second network. 8.The method of claim 2, wherein, The file generation node abnormality troubleshooting process is as follows: 1) service load check: First, the system or an operation and maintenance personnel needs to check the running state of the cross-network data exchange service responsible for file generation; the resource utilization rate index is mainly evaluated; if it is found that the CPU or memory utilization rate exceeds the preset threshold, it is determined that there is a resource bottleneck, and service expansion operation should be performed immediately to improve the service processing capacity; if the resource utilization rate is normal, proceed to the next step; 2) storage I / O performance check: after confirming that the computing resources are normal, the input / output performance of the underlying storage system needs to be diagnosed; a professional disk performance test tool is used for detection; if the detection result shows that the IOPS is lower than the specified standard, it indicates that there is a bottleneck in the storage write speed, and further in-depth troubleshooting is needed; if the I / O performance test result is normal, turn to other possibilities; 3) hard disk state and file lock check: Check the hard disk status to confirm whether there are physical bad tracks or logical errors in the storage hard disk, and ensure the health of the hardware foundation; check file locks, as file lock conflicts are extremely likely to occur in concurrent writing scenarios; in addition, it is also necessary to check whether the target file is abnormally locked; If there is a file lock conflict, the strategy of forcibly releasing the file lock and triggering the retry mechanism is adopted to attempt to recover the file writing process; If there is no file lock exception, it indicates that the problem is more hidden; 4) Log analysis: When no obvious abnormalities are found at the hardware and resource levels, detailed logs of the cross-network data exchange service and the application system need to be analyzed in depth to find key information and locate the root cause. 9.The method of claim 2, wherein, The file acquisition node timeout troubleshooting process is as follows: 1) Triggering alarm and manual intervention: The monitoring system will first automatically generate a file acquisition node timeout alarm and notify the operation and maintenance personnel to manually intervene in processing through a predetermined channel; 2) Checking the exchange platform service status: After receiving the alarm, the operation and maintenance personnel check the status of the cross-network data exchange service on the external network side to confirm whether it is running normally, whether there is a process stall or service crash, etc.; 3) Verifying network connectivity: If the service status is normal, the network connectivity from the external network to the file public area needs to be verified to rule out network level isolation or interruption problems; the verification of connectivity includes checking whether the network link is smooth, whether DNS resolution is normal, and whether the firewall policy allows access; 4) Checking file directory permissions: If there is no abnormality in network connectivity, the access permissions of the external network exchange service account to the target file or the directory where the target file is located in the file public area need to be checked.

Images