Observable platform data accuracy verification automation testing method and system

Abstract

The application provides an observable platform data accuracy verification automation test method and system, a master node and a target node perform adaptive image loading and deployment of a flow container; through cooperative timing control, the target node synchronously performs automatic packet capture and flow flow, and outputs a pcap file; the pcap file is automatically analyzed and real index calculation is performed, and after alignment with observable platform data to be verified, each item is compared, process logs are written, and key step exceptions are bottomed out. The application solves the image loading compatibility problem under the domesticization architecture, ensures accurate alignment of the packet capture time window and the flow process, realizes accurate comparison of the real traffic index and the platform collection index, and guarantees the reproducibility and result reliability of the automation test process.

Description

Technical Field

[0001] This invention belongs to the field of information technology, specifically relating to an automated testing method and system for verifying the accuracy of observable platform data. Background Technology

[0002] Currently, cloud-native observability technology has fully entered the era of eBPF non-intrusive data collection. Products such as DeepFlow, Prometheus, Grafana, and cloud vendor observability platforms are widely used for traffic monitoring, performance analysis, and fault location of Kubernetes containers, microservices, and cloud hosts.

[0003] These types of platforms generally have the following characteristics: 1. Non-intrusive data acquisition based on eBPF / kernel probes; 2. Facing complex Kubernetes environments with multiple nodes, multiple tenants, and multiple network planes; 3. Provides core metrics such as throughput, latency, PPS, connection count, and abnormal traffic; 4. In the fields of finance, telecommunications, and information technology innovation, data must be absolutely accurate, traceable, and verifiable.

[0004] However, existing observable platforms only focus on data collection and display, lacking standardized, automated, and reproducible accuracy verification capabilities. Most platforms require manual verification, and while some have general automated verification tools, they still have certain shortcomings. These can be summarized as follows: 1. The accuracy of data collection cannot be verified: eBPF / kernel data collection depends on kernel version, network environment, agent deployment location, and timestamp alignment. Any environmental differences may lead to metric deviations, but existing observable platforms cannot provide third-party true value comparisons.

[0005] 2. Manual verification is extremely costly and cannot be reproduced: If the observable platform requires manual verification, the entire process of image loading, container deployment, traffic flow analysis, packet capture and analysis, data query and comparison must be completed manually. A single verification takes several hours and is prone to result distortion due to human error, making it unsuitable for R&D iteration, batch acceptance, and periodic inspection.

[0006] 3. Inconsistent statistical standards and lack of comparison criteria: Existing observable platforms use fixed time windows for aggregation, such as 10s / 30s / 60s, while real traffic consists of microsecond-level data packets. When manually analyzing packet capture data and platform-collected data, there is no unified time window alignment or indicator calculation rules, which can easily lead to comparison biases. It is impossible to objectively determine the accuracy of platform data, resulting in unreliable results.

[0007] 4. Lack of usable tools in domestic environments: Existing observable platforms are all geared towards x86 and standard Linux, and have not been adapted for domestic architectures. Compatibility issues are prone to occur in image loading, command execution, and other processes, and they cannot run in mixed versions of Kylin, Kunpeng, Phytium, and K8s.

[0008] 5. Multi-node clusters cannot be used for collaborative verification: Existing observable platforms collect data across nodes, containers, and network cards. Manual operation makes it difficult to unify the node environment and synchronize packet capture and flow analysis, resulting in unreproducible verification results and complete verification failure. Summary of the Invention

[0009] The purpose of this invention is to provide an automated testing method and system for verifying the accuracy of observable platform data, solve the image loading compatibility problem under the domestic architecture, ensure the precise alignment of the packet capture time window and the flow capture process, achieve accurate comparison between real traffic indicators and platform-collected indicators, and ensure the reproducibility of the automated testing process and the credibility of the results.

[0010] To achieve the above objectives, the technical solution of the present invention is as follows: An automated testing method for verifying the accuracy of observable platform data includes: S1. The master node and the target node perform adaptive image loading and deployment of the streaming container; S2. Through coordinated timing control, the target node synchronously and automatically captures packets and streams traffic, and outputs a pcap file; S3. Perform automatic analysis of pcap files and calculation of real indicators, and compare them item by item with the data of the observable platform to be verified. S4. Write process logs and provide backup for critical steps in case of anomalies.

[0011] Furthermore, the adaptive image loading and deployment of the streaming container described in step S1 includes: The system executes a pre-defined test script on the master node, reads the test configuration file, detects the connectivity with the target node, and identifies the Kubernetes version and architecture of both the local and target nodes. Select the appropriate streaming container image based on the architecture, adaptively select the loading tool based on the Kubernetes version, and complete the local / remote loading of the streaming container image on the local machine and the target node. Replace the variables in the streaming container configuration template according to the test configuration file, and automatically generate and deploy the streaming container on the local machine and the target node.

[0012] Furthermore, the connectivity with the target node is detected, including: Check if the node is reachable via ICMP; Check if the node's SSH port is open; Verify whether the node is in the Ready state; Check whether the node returns a healthy status; Login and operation permission verification; If any detection or verification fails, output a failure message and execute the fallback mode as configured.

[0013] Furthermore, identifying the Kubernetes version and architecture includes: When identifying the K8s version, only the major version and minor version are retained. If the acquisition fails, the K8s version specified in the test configuration file is read. During architecture identification, the extracted local and target node architectures are verified for consistency and then uniformly converted into standard Kubernetes image architecture names.

[0014] Furthermore, the streaming container image includes: Mirroring based on abnormal traffic ratio: Built-in abnormal traffic simulation logic outputs a fixed proportion of abnormal traffic to verify the accuracy of application abnormality ratio metrics collected by the observable platform. The application constructs a mirror image based on connection ratio traffic: initiating TCP connections, establishing new connections, and active connections at a fixed rate; controlling the number of connections per second, connection success rate, and concurrent connections; used to verify the accuracy of the measured platform's connection rate, number of active connections, and connection establishment success rate metrics. TCP retransmission ratio model: Controllable packet loss during transmission, triggering automatic kernel retransmission, and outputting a fixed retransmission rate; used to verify the accuracy of TCP retransmission rate and packet loss rate metrics collected by the observable platform. TCP zero-window traffic model: Simulates a full receiver buffer, sends a zero-window announcement, generates standard zero-window events and zero-window probes; used to verify the accuracy of the platform's TCP zero-window count and zero-window duration metrics. General benchmark traffic construction mirror: simulates HTTP protocol request streaming to generate traffic; used to verify the accuracy of platform traffic rate, sending transport layer payload rate, receiving transport layer payload rate, sending traffic rate, receiving traffic rate, TCP connection establishment latency, system latency, and application response latency metrics.

[0015] Furthermore, step S2 includes: S201. Multi-observation point environment verification: The master node automatically checks whether all target nodes are in the same initial state through the test script; S202. Send synchronization instructions to all observation points: The master node sends a unified start instruction to all target nodes. The start instruction includes: the timestamp of the start of the flow capture, the duration of the flow capture, the network card for packet capture, and the pcap output path. S203. Synchronous packet capture from multiple observation points: The packet capture process of all target nodes starts at the same time according to the packet capture start timestamp and continues to run until the packet capture ends; S204. Synchronous Start of Traffic Flow Capture: After packet capture starts, all target nodes immediately begin traffic capture synchronously; the traffic capture container images execute their respective traffic simulations. S205, End of traffic capture, synchronously stop packet capture: All traffic capture containers synchronously stop sending traffic, the master node sends a stop packet capture command to all target nodes, and all target nodes immediately and synchronously stop packet capture. S206. Output standard pcap file: Each target node outputs a standard libpcap format pcap file; S207. Multi-observation point pcap file return: The master node automatically reclaims the pcap files of all target nodes through the copy command.

[0016] Furthermore, step S3, which involves automatic analysis of the pcap file and calculation of the true metrics, includes: S301. Read all data packets in the PCAP file, extract the required basic indicator data, and store them in a list; S302. Convert the list into a pandas.DataFrame, a tabular data structure in the Pandas library, convert the timestamps to datetime format, and set them as indexes; S303. Resample according to the specified time window, aggregate the basic indicator data, rename the aggregated column, and delete the window with no data; S304. Add a new column: use the aggregated column of basic indicator data to calculate or convert the required indicator to form the required indicator column; S305. Returns a DataFrame containing the required index columns.

[0017] Furthermore, step S3, after aligning with the observable platform data to be verified, includes the following item-by-item comparison: S311. Time window alignment, ensuring consistent time ranges on both sides; S312. Data collection dimensions are aligned, and the data sources on both sides remain consistent. S313. Align the units of the indicators, and keep the units of the numbers on both sides consistent; S314. Statistical standards should be aligned, and calculation rules on both sides should be consistent. S315. Abnormal data cleaning and alignment, ensuring consistency in the number of data entries and time points on both sides.

[0018] Furthermore, the critical step exception handling in step S4 includes: Before executing each critical step, a dedicated fallback plan is automatically bound to that step through a test script; during execution, step anomalies are captured in real time and executed according to the dedicated fallback plan; the critical steps include those that prevent subsequent steps from continuing or that result in unreliable or unreproducible final verification results; the dedicated fallback plan includes one or more combinations of retry, skip, downgrade, and use of backup resources.

[0019] In another aspect, this invention also proposes an automated testing system for verifying the accuracy of observable platform data, comprising: Loading and Deployment Control Module: The master node and target nodes perform adaptive image loading and deployment of streaming containers; Synchronous packet capture and traffic stream processing module: Through coordinated timing control, it realizes synchronous and automated packet capture and traffic stream processing of target nodes, and outputs pcap files; Analysis, calculation, and comparison module: performs automatic analysis of pcap files and calculation of real indicators, and compares them item by item with the data from the observable platform to be verified; Log recording and exception handling module: write process logs and handle exceptions in key steps.

[0020] Compared with the prior art, the present invention has the following significant advantages: 1. This invention proposes an adaptive image loading logic based on K8s version / architecture to solve the image loading compatibility problem under domestic architecture; 2. This invention achieves automated packet capture and traffic stream timing control through multi-node collaboration, ensuring precise alignment between the packet capture time window and the stream extraction process; 3. This invention designs a standardized pcap file analysis and platform data collection rule mapping to achieve accurate comparison between real traffic metrics and platform-collected metrics; 4. This invention proposes a full-process log traceability and anomaly fallback mechanism to ensure the reproducibility of the automated testing process and the reliability of the results. Attached Figure Description

[0021] Figure 1 This is an overall flowchart of Embodiment 1 of the present invention; Figure 2 This is a flowchart of multi-observation point packet capture and traffic analysis in Embodiment 3 of the present invention; Figure 3 This is a schematic diagram of the system structure of Embodiment 6 of the present invention. Detailed Implementation

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0023] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings.

[0024] Example 1: This embodiment provides a detailed description and technical implementation of the automated testing method for verifying the accuracy of observable platform data proposed in this invention.

[0025] Existing technologies for observable platform data mainly focus on areas such as cloud-native monitoring and collection, network traffic generation, container stress testing, and Kubernetes automated deployment. Their technical solutions generally focus on functions such as traffic generation, indicator collection, visualization, and general automated testing, but none of them address the core technical issue of verifying the accuracy of observable platform data.

[0026] This embodiment automatically completes the entire process of image loading, container deployment, traffic interception, packet capture analysis, and data query and comparison; the basic environment for implementation is as follows: Hardware environment: A multi-node K8s cluster based on the Galaxy Kylin operating system, supporting x86 / arm64 domestic architecture; Software environment: K8s (v1.x series), docker / containerd, tcpdump, pcap analysis script, kubectl; Configuration dependencies: config.ini configures node information, password, etc., and the image directory stores images adapted to different architectures.

[0027] like Figure 1 As shown, the automated testing method for verifying the accuracy of observable platform data in this embodiment includes: S1. The master node and the target node perform adaptive image loading and deployment of the streaming container.

[0028] This step mainly completes image loading and container deployment, specifically including: 1.1 Execute the preset test script on the master node, read the test configuration file, detect the connectivity with the target node, and identify the Kubernetes version and architecture of the local and target nodes; The test script contains the complete logic for automated testing, including reading configuration, detecting node connectivity, identifying version and architecture, and the logic for the master node to initialize and execute the test script.

[0029] The test configuration file is a parameter configuration file specifically for automated traffic testing. It includes the target node IP list for traffic capture / packet sniffing, the kubectl configuration file path, the traffic capture duration, the traffic type, the temporary storage path of the pcap file on the node, the local path of the metric analysis script, and the time window for data comparison.

[0030] If there is a problem with the connectivity between the master node and the target node, it will directly affect the subsequent packet capture process. Therefore, it is necessary to check the connectivity between the master node and the target node.

[0031] Identifying the Kubernetes version and architecture of the local and target nodes is necessary for installing the streaming container image on both the master and target nodes.

[0032] 1.2. Select the corresponding streaming container image based on the architecture, and adaptively select the loading tool according to the K8s version to complete the local / remote loading of the streaming container image on the local machine and the target node.

[0033] Different nodes have different CPU architectures, requiring different image files. Identifying whether the architecture is x86 or ARM64 allows you to filter the image for x86-specific or ARM64-specific files, preventing installation failures.

[0034] Before and after Kubernetes version 1.24, the command-line tools for loading images differed; after version 1.24, ctr became the mainstream tool. Therefore, once the version is identified, the corresponding command-line tool (docker or ctr) can be automatically selected to complete the image loading operation, preventing command execution failures due to version differences.

[0035] Based on the Kubernetes version and architecture of the master node and target nodes, the corresponding architecture image file is automatically selected from the image directory. Through the adaptive selection of the loading tool, the local loading of the streaming container image of the master node and the remote loading of the streaming container images of each target node are completed.

[0036] 1.3 Replace the variables in the streaming container configuration template according to the test configuration file, and automatically generate and deploy the streaming container on the local machine and the target node.

[0037] The parameters and variables configured in the streaming container image are general values ​​and need to be replaced according to the actual test scenario. After replacement, the streaming container is generated and deployed.

[0038] S2. Through coordinated timing control, the target node can synchronously and automatically capture packets and stream traffic, and output a pcap file.

[0039] This step mainly completes the synchronous automated traffic streaming and packet capture, including the following: 2.1 Multi-observation point environment verification: The master node automatically checks whether all target nodes are in the same initial state through the test script; 2.2. Send synchronization instructions to all observation points: The master control node sends a unified start instruction to all target nodes. The start instruction includes: the timestamp of the start of the flow capture, the duration of the flow capture, the network card for packet capture, and the pcap output path. 2.3 Synchronous packet capture from multiple observation points: The packet capture process of all target nodes starts at the same time according to the packet capture start timestamp and continues to run until the packet capture ends; 2.4 Synchronous Start of Traffic Flow Capture: After packet capture starts, all target nodes immediately begin traffic capture synchronously; the traffic capture container images execute their respective traffic simulations; 2.5. Streaming ends, packet capture stops synchronously: All streaming containers stop sending traffic synchronously, the master node sends a stop packet capture command to all target nodes, and all target nodes immediately stop packet capture synchronously; 2.6 Output standard pcap file: Each target node outputs a standard libpcap format pcap file; 2.7 Multi-observation point pcap file return: The master node automatically reclaims the pcap files of all target nodes through the copy command.

[0040] S3. Perform automatic analysis of pcap files and calculation of real indicators, and compare them item by item after aligning them with the data from the observable platform to be verified.

[0041] This step mainly involves packet capture analysis, data querying, and comparison.

[0042] The automatic analysis and calculation of true metrics for pcap files includes the following: 3.1 Read all data packets in the PCAP file, extract the required basic indicator data, and store them in a list; 3.2. Convert the list into a pandas.DataFrame, a tabular data structure from the Pandas library, convert the timestamps to datetime format, and set them as indexes; 3.3 Resample according to the specified time window, aggregate the basic indicator data, rename the aggregated column, and delete the window with no data; 3.4 Add a new column: use the aggregated column of basic indicator data to calculate or convert the required indicators to form the required indicator column; 3.5 Returns a DataFrame containing the required index columns.

[0043] The process of aligning with the data from the observable platform to be verified includes: Time windows are aligned to ensure consistent time ranges on both sides; data collection dimensions are aligned to ensure consistent data sources on both sides; indicator units are aligned to ensure consistent numerical units on both sides; statistical definitions are aligned to ensure consistent calculation rules on both sides; and abnormal data cleaning is aligned to ensure consistent number of data entries and time points on both sides.

[0044] S4. Write process logs and provide backup for critical steps in case of anomalies.

[0045] Before executing each critical step, a dedicated fallback plan is automatically bound to that step through a test script; during execution, step anomalies are captured in real time and executed according to the dedicated fallback plan; the critical steps include those that prevent subsequent steps from continuing or that result in unreliable or unreproducible final verification results; the dedicated fallback plan includes one or more combinations of retry, skip, downgrade, and use of backup resources.

[0046] The method described in this embodiment solves the image loading compatibility problem under the domestic architecture, ensures that the packet capture time window and the traffic acquisition process are precisely aligned, achieves accurate comparison between real traffic indicators and platform-collected indicators, and ensures the reproducibility of the automated testing process and the credibility of the results.

[0047] Example 2: This embodiment is a further detailed explanation of step S1 in embodiment 1.

[0048] In this embodiment, the preset test script is run.sh. In addition to the execution logic of reading configuration, detecting node connectivity, and identifying version and architecture in this step, run.sh also has the core logic of subsequent steps, including basic environment verification, command line parameter parsing, log initialization, K8s node SSH permissions and kubectl operation permissions, as well as fallback logic for initialization failures.

[0049] The logic for reading the test configuration file via the test script `run.sh` involves using a base template, such as `test_config.yaml`, written by testers in YAML / INI / JSON format, specifically for automated traffic testing and tailored to the test scenario. A configuration template tool is then provided, allowing input of core parameters such as node IPs and traffic duration to automatically generate the configuration file. If integrated with a testing platform, the platform can automatically distribute the configuration file to the execution machine based on the test task parameters. The default path for the test configuration file can be a subdirectory of `config` at the same level as `run.sh`, or a custom path can be used.

[0050] The method used in this step to detect node connectivity is a hierarchical detection logic, and the steps are as follows: Basic network reachability: The target node is reachable via ICMP as confirmed by ping. Service port reachability: Probe whether SSH port (22) is open via telnet; verify whether the nodes in the K8s cluster are in the Ready state via kubectl; check whether the target node returns a healthy state via curl; Operation permission verification: Verify whether SSH passwordless login is effective; verify container operation permissions via kubectl.

[0051] If any detection or verification fails, output the failure information and execute the fallback mode according to the configuration: output node IP and failure type; support two modes: skipping failed nodes or terminating the process, the mode is specified through the test configuration file.

[0052] In this step, when identifying the Kubernetes version, first log in to the corresponding nodes (including the master node and the target node) and extract the version using grep. Only the major and minor versions are retained during extraction; for example, version v1.24.6 is extracted as 1.24. If extraction fails, the Kubernetes version manually specified in the test configuration file is read.

[0053] When performing architecture identification, you can use `uname -m` to identify the CPU architecture of the master node first, then the target node, and verify consistency. If the CPU architectures of the master node and the target node are different, corresponding streaming container images need to be prepared separately. The identified architectures also need to be uniformly converted to the standard Kubernetes image architecture. For example, if x86_64 is identified, it should be converted to amd64; if aarch64 is identified, it should be converted to arm64. The purpose of the conversion is to ensure that the identified architectures are consistent with the architecture standards of the Kubernetes image repository, avoiding image pull failures.

[0054] In this step, the deployed streaming container image should include at least the following: (1) Construct an image using abnormal ratio traffic (app-abnormal-ratio-arm-v4.tar): Function: Construct a mirror image using an abnormal proportion of traffic; Built-in abnormal traffic simulation logic: timeout, rejection, reset, failed request; Output a fixed percentage of abnormal traffic (e.g., 10% / 20% / 30% / 66.67%). Provide the observable platform with the known true anomaly rate to verify the accuracy of the platform's "application anomaly ratio" indicator.

[0055] (2) Construct a mirror image using the application connection ratio traffic (app-connect-ratio-arm / x86-v1.tar): Function: Construct a mirror based on the proportion of application connection traffic; Initiating TCP connections, establishing new connections, and activating new connections at a fixed rate; Control the number of connections per second, connection success rate, and concurrent connections; Used to verify the accuracy of platform connection speed, number of active connections, and connection success rate metrics; (3) Constructing a mirror image using the TCP retransmission ratio (tcp-retransmission-ratio-arm / x86-v4.tar): Function: Construct a mirror image based on the TCP retransmission ratio; Controllable packet loss during transmission, triggering automatic retransmission by the kernel; Output traffic with a fixed retransmission rate (e.g., 1% / 5% / 10%). Used to verify the accuracy of the platform's TCP retransmission rate and packet loss rate data collection; (4) Constructing a TCP zero-window traffic image (tcp-zero-window-arm / x86-v2.tar): Function: Construct a TCP mirror using zero-window traffic; Simulates a full receiver buffer and sends a zero-window announcement; Generate standard zero-window events and zero-window probes; Used to verify the accuracy of the platform's TCP zero-window count and zero-window duration collection; (5) Constructing a general baseline traffic image (test-pcap-arm / x86.tar): Functions: Constructing a mirror image based on traffic rate, transmit transport layer payload rate, receive transport layer payload rate, transmit traffic rate, receive traffic rate, TCP connection establishment delay, system delay, and application response delay; Simulate HTTP protocol requests to generate traffic; Used to verify the accuracy of platform traffic rate, transmit transport layer payload rate, receive transport layer payload rate, transmit traffic rate, receive traffic rate, TCP connection establishment delay, system delay, and application response delay data collection.

[0056] The deployment process involves five mirrors, with the YAML template being replaced automatically first.

[0057] For example, the following command can be used: "cp traffic-pod-template.yaml traffic-pod-final.yaml sed -i "s / {{NODE_NAME}} / $NODE_NAME / g" traffic-pod-final.yaml sed -i "s / {{IMAGE_ABNORMAL}} / $IMAGE_ABNORMAL / g" traffic-pod-final.yaml sed -i "s / {{IMAGE_CONNECT}} / $IMAGE_CONNECT / g" traffic-pod-final.yaml sed -i "s / {{IMAGE_RETRANS}} / $IMAGE_RETRANS / g" traffic-pod-final.yaml sed -i "s / {{IMAGE_ZEROWIN}} / $IMAGE_ZEROWIN / g" traffic-pod-final.yaml sed -i "s / {{IMAGE_PCAP}} / $IMAGE_PCAP / g" traffic-pod-final.yaml"; This command is based on a predefined YAML template file traffic-pod-template.yaml. It replaces the placeholders (such as {{NODE_NAME}}, {{IMAGE_ABNORMAL}}, etc.) with the actual values ​​of the script runtime environment variables, and finally generates a complete YAML file traffic-pod-final.yaml that can be directly submitted to Kubernetes deployment.

[0058] Then, using docker / ctr adaptive mode, the image is automatically loaded and deployed to the specified target node, waiting for the pod to be ready, and verifying whether the deployment was successful.

[0059] This embodiment describes in detail the adaptive image loading logic based on K8s version / architecture, which can solve the image loading compatibility problem under the domestic architecture.

[0060] Example 3: This embodiment is a further detailed explanation of step S2 in embodiment 1.

[0061] Step S2, multi-observation point packet capture and traffic stream analysis, is the core of the automated verification scheme of this invention. Its purpose is to simultaneously start packet capture on multiple target nodes in the K8s cluster, synchronously start stream analysis, synchronously stop packet capture after stream analysis ends, and finally output a standard pcap traffic file as a truth benchmark for verifying the accuracy of platform data.

[0062] The entire process strictly follows the procedure: synchronously start packet capture, execute traffic streaming, synchronously stop packet capture, and output a pcap file. Detailed execution steps are as follows: Figure 2 As shown, it includes the following: Step 1: Confirm the environment at multiple observation points.

[0063] Before performing stream interception and packet capture, the following steps are performed automatically using a test script (such as run.sh): 1. Read the list of all nodes to be tested from the configuration file; 2. Check that all nodes have deployed 5 streaming containers; 3. Check node time synchronization and NTP alignment to ensure consistent time across multiple nodes; 4. Check if the / pcap shared directory is readable and writable; 5. Purpose: To ensure that all observation points are in the same initial state and can perform actions synchronously.

[0064] Step 2: Send synchronization commands to all observation points.

[0065] The master node sends a unified startup command to all streaming nodes via SSH / kubectlexec.

[0066] 1. The instructions include: 2. Set the start timestamp of the stream; 3. Duration of the burst (e.g., 60 seconds); 4. Capture network packets from network interface cards (NICs), such as any / eth0 / service NICs, etc. 5. pcap output path.

[0067] All target nodes receive the same instructions, ensuring synchronized actions.

[0068] Step 3: Simultaneously start packet capture at multiple observation points.

[0069] The command is: tcpdump -I any -w / pcap / result.pcap -U.

[0070] Key points: 1. Synchronous startup: The packet capture process on all target nodes starts at the same time; 2. -iany: Capture all network interface traffic; 3. -w: Output the raw pcap file; 4. -U: Write to the file in real time, without caching; 5. Runs in the background and does not block streaming.

[0071] The packet capture process will continue to run until the packet capture ends.

[0072] Step 4: Simultaneously start traffic generation.

[0073] Once packet capture begins, all nodes immediately synchronize and start capturing the data stream: Each of the five containers performs its own traffic simulation: 1. app-abnormal-ratio: Generates a fixed ratio of abnormal requests; 2. app-connect-ratio: Generates a fixed-rate TCP connection; 3. tcp-retransmission-ratio: Generates controllable TCP retransmission traffic; 4. tcp-zero-window: Generates TCP zero-window traffic; 5. test-pcap: Continuous packet capture without interruption; The traffic interception process runs continuously according to the configured time, with all nodes exhibiting identical traffic type, intensity, and duration. No additional interfering traffic is generated during the traffic interception process.

[0074] Step 5: Once the packet capture is complete, stop capturing packets synchronously.

[0075] When the countdown to the finale ends: 1. All streaming containers synchronously stop sending traffic; 2. The master node sends a stop packet capture command to all observation points; 3. The tcpdump process on all nodes exits synchronously; Step 6: Output a standard pcap file.

[0076] After packet capture stops: 1. Output a result.pcap for each observation point (target node); 2. File path: / pcap / result.pcap; 3. File format: Standard libpcap format, which can be parsed by Wireshark, tcptrace, and tcpstat; 4. Includes complete traffic information: timestamp, source and destination IPs, port, protocol, packet sequence, retransmission, zero window, abnormal connection, etc.

[0077] Step 7: Send the pcap file back to the master node.

[0078] The master node automatically reclaims the pcap of all observation points in the following manner; kubectlcpdefault / traffic-pod: / pcap / result.pcap. / result / node-xxx.pcap; These pcap files are the source of truth for verifying platform metrics.

[0079] This embodiment specifically implements automated packet capture and traffic bursting timing control with multi-node collaboration to ensure precise alignment between the packet capture time window and the bursting process.

[0080] Example 4: This embodiment is a further detailed explanation of step S3 in embodiment 1.

[0081] Step S3, pcap file analysis and true metric calculation, includes: In this embodiment, multiple pcap analysis scripts are provided for pcap file analysis. Taking one of these analysis scripts, pcap_speed.py, as an example: Script function: Receives the PCAP file path and time window size, and returns traffic statistics aggregated by window, including total bytes and Kbps; The parameters include: pcap_file: String, the path to the PCAP file; window: Integer, the size of the time window (in seconds), the default value is 1 second.

[0082] Execution steps: 1. Read and parse data packets; Use rdpcap(pcap_file) to read all packets from the PCAP file; Iterate through each data packet, extract the timestamp (converted to floating-point) and packet_size (data packet size in bytes), and store them in a list; If the list is empty (no data packet), a ValueError exception is thrown.

[0083] 2. Data format conversion and index settings; Convert the list to a pandas.DataFrame with columns named timestamp and size; Convert the timestamp (in seconds) to datetime format and set it as a DataFrame index (to facilitate time series resampling).

[0084] 3. Aggregate by time window; Use resample(f"{window}S") to resample by the specified window and aggregate the size column (sum, i.e., the total number of bytes per window). Rename the aggregate column to total_bytes and delete the window with no data (dropna()).

[0085] 4. Calculate kbps; In the field of communications, the conversion standard is: 1 Byte = 8 bits, 1 Kbps = 1000 bits / second; Formula: kbps = (total bytes × 8) / (1000 × window size), add a kbps column to store the results.

[0086] 5. Return value; A DataFrame containing total_bytes (total number of bytes) and kbps (traffic rate).

[0087] 6. Usage method; python3 pcap_speed.py <PCAP file path> The analysis of other pcap analysis scripts and the real indicator calculation process are similar to this.

[0088] The alignment with the data of the observable platform to be verified in step S3 specifically includes: 1. Time window alignment.

[0089] The start time and end time of pcap capture; Strictly keep consistent with the query time range of the data collected by the platform (error ≤ 1 second).

[0090] Example: If packets are captured from 10:00:00 to 10:01:00, the platform must query the data from 10:00:00 to 10:01:00.

[0091] 2. Collection dimension alignment.

[0092] Make the platform data and the real data come from exactly the same observation object.

[0093] The alignment content includes: Node IP alignment (the same node); Network card name alignment (the same network card); Container name / ID alignment (the same container); Protocol type alignment (TCP / UDP / ICMP); Port alignment (the same service port); Direction alignment (inbound / outbound are consistent); Purpose: Ensure that the data sources on both sides are exactly the same and comparable.

[0094] 3. Indicator unit alignment.

[0095] The units of the platform and the pcap true value must be completely consistent: Throughput: uniformly use Mbps; Latency: uniformly use ms; Packet loss rate / retransmission rate / anomaly rate: uniformly use % (percentage, 2 decimal places); Number of connections: uniformly use count / second; Example: If the platform returns KB / s, it is automatically converted to Mbps; if the platform returns μs, it is automatically converted to ms; Ensure that the numerical units on both sides are the same before comparison.

[0096] 4. Align statistical standards.

[0097] The calculation rules on both sides must be exactly the same; otherwise, the comparison is meaningless.

[0098] The unified standard in this embodiment is as follows: Retransmission rate = Number of retransmitted packets ÷ Total number of packets sent × 100%; Throughput = Total bytes × 8 ÷ Time ÷ 1024 ÷ 1024 (Mbps); Connection success rate = Number of successful connections ÷ Number of connection requests × 100%; Application anomaly rate = (Number of abnormal packets ÷ Total number of packets) × 100%; Zero window count = total number of zero window packets captured; The platform must use the exact same calculation formula as the pcap analysis script.

[0099] 5. Clean and align abnormal data.

[0100] Automatically clean up invalid data from both sides to maintain consistency: Remove null values, 0 values, and values ​​that failed to be collected; Remove abnormal spike values ​​that exceed a reasonable range (such as latency > 5000ms); Missing points are filled using linear interpolation or the mean of adjacent points; Only retain valid time slice data; After cleaning, the number of data entries and time points on both sides are completely consistent.

[0101] This embodiment specifically designs a standardized pcap file analysis and platform data collection rule mapping to achieve accurate comparison between real traffic metrics and platform-collected metrics.

[0102] Example 5: This embodiment further elaborates on the key step exception handling described in step S4 of embodiment 1.

[0103] Critical step exception fallback means that in the entire automated testing process, if an exception occurs at any step, the system will not crash or be interrupted directly. Instead, it will automatically retry, skip, downgrade, alert, or log according to preset strategies to ensure that the overall process continues to execute to the end.

[0104] 1. Before executing each step, register an exception fallback strategy: Before executing each critical step, the script automatically assigns a specific fallback plan to that step.

[0105] For example: Configuration read error -> Load default configuration; If a node is not reachable, skip that node. Image loading failed -> Retry 3 times; Pod startup failed -> delete and rebuild; Packet capture failed -> Preserve the current state without affecting other nodes; Analysis failed -> Log the error and continue comparing with other indicators; 2. During execution: Real-time exception capture: The script enables exception handling for each critical step: The command returns a non-zero value; No response after timeout; The file does not exist; Service unavailable; Insufficient permissions; Container startup failed; The packet capture file is empty; Platform interface failed; Once captured, the fallback process begins immediately.

[0106] 3. After an anomaly occurs: Implement a fallback measure according to the severity level. (1) Configure a fallback mechanism for read exceptions; Errors: Missing configuration file, incorrect format, insufficient permissions - fallback steps; Immediately load the default configuration file; Log warnings; Continue executing the process without interruption; (2) Fallback for abnormal node connectivity; Errors: Unable to ping, unable to SSH, node NotReady - fallback steps; Mark this node as unavailable; Output exception logs; Automatically skip this node; Other normal nodes continue execution; The process will not be interrupted; (3) Fallback for abnormal image loading; Errors: Image not found, loading failed, repository pull failed - fallback steps; Automatically retry 1 to 3 times (with a 3-second interval); Retry failed -> Load using local offline tar package; Still failed -> Use the minimized fallback image (busybox / test-pcap). Record the downgrade log; Continue executing the process; (4) YAML generation / deployment exception fallback; Errors: Template error, missing variables, Pod startup failure, image not found (fallback step); Output error log; Delete the abnormal Pod; Regenerate the YAML and redeploy; If the retries fail after two attempts, skip this node. The main process continues to execute; (5) Coverage for abnormal traffic / packet capture; Abnormalities: The packet capture process exits, no data is captured, and tcpdump fails. (Follow-up steps for abnormalities) Restart the packet capture / streaming process; Extend the burst time (+30 seconds); Still failed -> Preserve the pcap scene; Mark this node as having packet capture errors; The process continues without affecting other nodes; (6) Pcap analysis script exception fallback; Errors: Script execution failed, indicators are empty, pcap file is corrupted (fallback steps); Rerun the analysis script; Failure -> Switch to the alternative analysis script (the alternative script in extract-metric-data-scripts). Still failed -> Mark the indicator as abnormal; Skip this metric and continue comparing other metrics; The main process remains uninterrupted; (7) Platform data query anomaly fallback; Errors: Interface timeout, no data, authentication failure, incorrect return format - fallback steps; Automatically retry the query (3 times); Failure -> Expand the time window and query again; Still failing -> Use historical benchmarks or mark the metric as uncomparable; Continue to compare with other available metrics; The process will not be interrupted; (8) Data comparison / calculation deviation rate anomaly fallback; Errors: Data format mismatch, missing values, calculation errors (fallback steps); Automatic re-normalization and alignment; Clean up invalid data; Still failed -> Skip this metric comparison; Output an exception description; Continue generating the final report; 4. After the fallback is completed: Unify logging and status marking: After each fallback, the system automatically records: the abnormal step, the abnormal time, the abnormal node, the abnormal reason, the fallback strategy executed, and the final status (success / degradation / skip); and generates a status file.

[0107] In summary, this embodiment ensures uninterrupted process through a full-process hierarchical exception fallback mechanism, the specific process of which is as follows: Pre-defined fallback strategy: Before executing each critical step, predefine the handling method (retry, skip, degrade, use backup resources) in case of an exception in that step.

[0108] Real-time anomaly capture: Real-time anomaly capture is performed on key steps such as configuration reading, node detection, image loading, container deployment, traffic streaming, packet capture, pcap analysis, platform data query, and data comparison.

[0109] Tiered bottom-line treatment: Configuration error -> Load default configuration; Node exception -> Mark and skip; Image error -> Automatic retry, if it fails, use the fallback image; If deployment fails, delete and rebuild; if retry fails, skip the node. Streaming / packet capture error -> Restart process or extend time; Analyze the anomaly -> Re-analyze using a backup script; Platform query error -> Retry or extend the query time window; Compare anomalies -> Clean the data and then realign.

[0110] Unified logging: All exceptions and fallback actions are written to the log for easy traceability.

[0111] Uninterrupted process guarantee: An anomaly in a single step or node will not affect the overall process. The system will automatically skip or downgrade the process and continue execution until the final test report is generated.

[0112] Through the above mechanism, the goal of ensuring that no abnormality will interrupt the automated testing process can be achieved.

[0113] Example 6: This embodiment proposes an automated testing system for verifying the accuracy of observable platform data, such as... Figure 3 As shown, it includes: Loading and Deployment Control Module: The master node and target nodes perform adaptive image loading and deployment of streaming containers; Synchronous packet capture and traffic stream processing module: Through coordinated timing control, it realizes synchronous and automated packet capture and traffic stream processing of target nodes, and outputs pcap files; Analysis, calculation, and comparison module: performs automatic analysis of pcap files and calculation of real indicators, and compares them item by item with the data from the observable platform to be verified; Log recording and exception handling module: write process logs and handle exceptions in key steps.

[0114] The adaptive image loading and deployment of the streaming container in the loading and deployment control module includes: The system executes a pre-defined test script on the master node, reads the test configuration file, detects the connectivity with the target node, and identifies the Kubernetes version and architecture of both the local and target nodes. Select the appropriate streaming container image based on the architecture, adaptively select the loading tool based on the Kubernetes version, and complete the local / remote loading of the streaming container image on the local machine and the target node. Replace the variables in the streaming container configuration template according to the test configuration file, and automatically generate and deploy the streaming container on the local machine and the target node.

[0115] Detecting connectivity with the target node includes: Check if the node is reachable via ICMP; Check if the node's SSH port is open; Verify whether the node is in the Ready state; Check whether the node returns a healthy status; Login and operation permission verification; If any detection or verification fails, output a failure message and execute the fallback mode as configured.

[0116] Identifying the Kubernetes version and architecture includes: When identifying the K8s version, only the major version and minor version are retained. If the acquisition fails, the K8s version specified in the test configuration file is read. During architecture identification, the extracted local and target node architectures are verified for consistency and then uniformly converted into standard Kubernetes image architecture names.

[0117] The streaming container image includes: Mirroring based on abnormal traffic ratio: Built-in abnormal traffic simulation logic outputs a fixed proportion of abnormal traffic to verify the accuracy of application abnormality ratio metrics collected by the observable platform. The application constructs a mirror image based on connection ratio traffic: initiating TCP connections, establishing new connections, and active connections at a fixed rate; controlling the number of connections per second, connection success rate, and concurrent connections; used to verify the accuracy of the measured platform's connection rate, number of active connections, and connection establishment success rate metrics. TCP retransmission ratio model: Controllable packet loss during transmission, triggering automatic kernel retransmission, and outputting a fixed retransmission rate; used to verify the accuracy of TCP retransmission rate and packet loss rate metrics collected by the observable platform. TCP zero-window traffic model: Simulates a full receiver buffer, sends a zero-window announcement, generates standard zero-window events and zero-window probes; used to verify the accuracy of the platform's TCP zero-window count and zero-window duration metrics. General benchmark traffic construction mirror: simulates HTTP protocol request streaming to generate traffic; used to verify the accuracy of platform traffic rate, sending transport layer payload rate, receiving transport layer payload rate, sending traffic rate, receiving traffic rate, TCP connection establishment latency, system latency, and application response latency metrics.

[0118] The synchronous packet capture and streaming module includes: Multi-observation point environment verification: The master node automatically checks whether all target nodes are in the same initial state through the test script; Synchronization command is sent to all observation points: The master control node sends a unified start command to all target nodes. The start command includes: the start timestamp of the flow capture, the duration of the flow capture, the network card for packet capture, and the pcap output path. Synchronous packet capture from multiple observation points: The packet capture process of all target nodes starts at the same time according to the packet capture start timestamp and continues to run until the packet capture ends; Synchronous traffic capture: After packet capture starts, all target nodes immediately begin traffic capture synchronously; the traffic capture container images execute their respective traffic simulations. When the streaming ends, packet capture stops synchronously: all streaming containers stop sending traffic synchronously, the master node sends a stop packet capture command to all target nodes, and all target nodes immediately stop packet capture synchronously. Output standard pcap file: Each target node outputs a standard libpcap format pcap file; Multi-observation point pcap file return: The master node automatically reclaims the pcap files of all target nodes through the copy command.

[0119] The automatic analysis and calculation of pcap files and the calculation of true indicators in the analysis, calculation, and comparison module include: Read all data packets in the PCAP file, extract the required basic indicator data, and store them in a list; Convert the list into a pandas.DataFrame, a tabular data structure in the Pandas library, convert the timestamps to datetime format, and set them as indexes; Resample according to the specified time window, aggregate basic indicator data, rename the aggregated column, and delete windows with no data; Add a new column, which uses the aggregated column of basic indicator data to calculate or convert the required indicator to form the required indicator column; Returns a DataFrame containing the desired index columns.

[0120] The item-by-item comparison after alignment with the observable platform data to be verified includes: Time windows are aligned, ensuring that the time ranges on both sides remain consistent. Data collection dimensions are aligned, and data sources on both sides remain consistent. The units of the indicators are aligned, and the units of the numbers on both sides are consistent. Statistical definitions should be aligned, and calculation rules should be consistent on both sides; Abnormal data is cleaned and aligned to ensure consistency in the number of data entries and time points on both sides.

[0121] The critical steps and exception handling in the log recording and exception handling module include: Before executing each critical step, a dedicated fallback plan is automatically bound to that step through a test script; during execution, step anomalies are captured in real time and executed according to the dedicated fallback plan; the critical steps include those that prevent subsequent steps from continuing or that result in unreliable or unreproducible final verification results; the dedicated fallback plan includes one or more combinations of retry, skip, downgrade, and use of backup resources.

[0122] The automated testing system for verifying the accuracy of observable platform data proposed in this embodiment can achieve the automated testing methods for verifying the accuracy of observable platform data described in embodiments 1, 2, 3, 4, and 5, and has the same technical effect.

[0123] The above-described embodiments are merely preferred embodiments of the present invention and are only used to help understand the method and core ideas of this application. The scope of protection of the present invention is not limited to the above embodiments, and all technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An automated testing method for verifying the accuracy of observable platform data, characterized in that, include: S1. The master node and the target node perform adaptive image loading and deployment of the streaming container; S2. Through coordinated timing control, synchronous and automated packet capture and traffic streaming are achieved at the target node, and a pcap file is output. S3. Perform automatic analysis of pcap files and calculation of real indicators, and compare them item by item after aligning them with the data of the observable platform to be verified. S4. Write process logs and provide backup for critical steps in case of anomalies. The critical step exception handling includes: before executing each critical step, automatically binding a dedicated exception handling scheme to the step through a test script; during execution, step exceptions are captured in real time and executed according to the dedicated exception handling scheme; the critical steps include steps that prevent subsequent steps from continuing, or steps that make the final verification result unreliable or unreproducible; the dedicated exception handling scheme includes one or more combinations of retry, skip, downgrade, and use of backup resources.

2. The automated testing method for verifying the accuracy of observable platform data according to claim 1, characterized in that, The adaptive image loading and deployment of the streaming container described in step S1 includes: The system executes a pre-defined test script on the master node, reads the test configuration file, detects the connectivity with the target node, and identifies the Kubernetes version and architecture of both the local and target nodes. Select the appropriate streaming container image based on the architecture, adaptively select the loading tool based on the Kubernetes version, and complete the local / remote loading of the streaming container image on the local machine and the target node. Replace the variables in the streaming container configuration template according to the test configuration file, and automatically generate and deploy the streaming container on the local machine and the target node.

3. The automated testing method for verifying the accuracy of observable platform data according to claim 2, characterized in that, Detecting connectivity with the target node includes: Check if the node is reachable via ICMP; Check if the node's SSH port is open; Verify whether the node is in the Ready state; Check whether the node returns a healthy status; Login and operation permission verification; If any detection or verification fails, output a failure message and execute the fallback mode as configured.

4. The automated testing method for verifying the accuracy of observable platform data according to claim 2, characterized in that, Identifying the Kubernetes version and architecture includes: When identifying the K8s version, only the major version and minor version are retained. If the acquisition fails, the K8s version specified in the test configuration file is read. During architecture identification, the extracted local and target node architectures are verified for consistency and then uniformly converted into standard Kubernetes image architecture names.

5. The automated testing method for verifying the accuracy of observable platform data according to claim 2, characterized in that, The streaming container image includes: Mirroring based on abnormal traffic ratio: Built-in abnormal traffic simulation logic outputs a fixed proportion of abnormal traffic to verify the accuracy of application abnormality ratio metrics collected by the observable platform. The application constructs a mirror image based on connection ratio traffic: initiating TCP connections, establishing new connections, and active connections at a fixed rate; controlling the number of connections per second, connection success rate, and concurrent connections; used to verify the accuracy of the measured platform's connection rate, number of active connections, and connection establishment success rate metrics. TCP retransmission ratio model: controllable packet loss during transmission, triggering automatic kernel retransmission, and outputting a fixed retransmission rate; used to verify the accuracy of TCP retransmission rate and packet loss rate metrics collected by the observable platform. TCP zero-window traffic model: Simulates a full receiver buffer, sends a zero-window announcement, generates standard zero-window events and zero-window probes; used to verify the accuracy of the platform's TCP zero-window count and zero-window duration metrics. General benchmark traffic construction mirror: simulates HTTP protocol request streaming to generate traffic; used to verify the accuracy of platform traffic rate, sending transport layer payload rate, receiving transport layer payload rate, sending traffic rate, receiving traffic rate, TCP connection establishment latency, system latency, and application response latency metrics.

6. The automated testing method for verifying the accuracy of observable platform data according to claim 1, characterized in that, Step S2 includes: S201. Multi-observation point environment verification: The master node automatically checks whether all target nodes are in the same initial state through the test script; S202. Send synchronization instructions to all observation points: The master node sends a unified start instruction to all target nodes. The start instruction includes: the timestamp of the start of the flow capture, the duration of the flow capture, the network card for packet capture, and the pcap output path. S203. Synchronous packet capture from multiple observation points: The packet capture process of all target nodes starts at the same time according to the packet capture start timestamp and continues to run until the packet capture ends; S204. Synchronous Start of Traffic Flow Capture: After packet capture starts, all target nodes immediately begin traffic capture synchronously; the traffic capture container images execute their respective traffic simulations. S205, End of traffic capture, synchronously stop packet capture: All traffic capture containers synchronously stop sending traffic, the master node sends a stop packet capture command to all target nodes, and all target nodes immediately and synchronously stop packet capture. S206. Output standard pcap file: Each target node outputs a standard libpcap format pcap file; S207. Multi-observation point pcap file return: The master node automatically reclaims the pcap files of all target nodes through the copy command.

7. The automated testing method for verifying the accuracy of observable platform data according to claim 1, characterized in that, Step S3, which involves automatic analysis of the pcap file and calculation of the true metrics, includes: S301. Read all data packets in the PCAP file, extract the required basic indicator data, and store them in a list; S302. Convert the list into a pandas.DataFrame, a tabular data structure in the Pandas library, convert the timestamps to datetime format, and set them as indexes; S303. Resample according to the specified time window, aggregate the basic indicator data, rename the aggregated column, and delete the window with no data; S304. Add a new column: use the aggregated column of basic indicator data to calculate or convert the required indicator to form the required indicator column; S305. Returns a DataFrame containing the required index columns.

8. The automated testing method for verifying the accuracy of observable platform data according to claim 1, characterized in that, Step S3, after aligning with the observable platform data to be verified, includes the following item-by-item comparison: S311. Time window alignment, ensuring consistent time ranges on both sides; S312. Data collection dimensions are aligned, and the data sources on both sides remain consistent. S313. Align the units of the indicators, and keep the units of the numbers on both sides consistent; S314. Statistical standards should be aligned, and calculation rules on both sides should be consistent. S315. Abnormal data cleaning and alignment, ensuring consistency in the number of data entries and time points on both sides.

9. An automated testing system for verifying the accuracy of observable platform data, characterized in that, include: Loading and Deployment Control Module: The master node and target nodes perform adaptive image loading and deployment of streaming containers; Synchronous packet capture and traffic stream processing module: Through coordinated timing control, it realizes synchronous and automated packet capture and traffic stream processing of target nodes, and outputs pcap files; Analysis, calculation, and comparison module: performs automatic analysis of pcap files and calculation of real indicators, and compares them item by item with the data from the observable platform to be verified; Log recording and exception handling module: process log writing, exception handling for critical steps; The critical step exception handling includes: before executing each critical step, automatically binding a dedicated exception handling scheme to the step through a test script; during execution, step exceptions are captured in real time and executed according to the dedicated exception handling scheme; the critical steps include steps that prevent subsequent steps from continuing, or steps that make the final verification result unreliable or unreproducible; the dedicated exception handling scheme includes one or more combinations of retry, skip, downgrade, and use of backup resources.

Images