System-level data real-time acquisition and downloading system based on HTTP streaming transmission

Abstract

The invention is applicable to the technical field of data transmission, and particularly relates to a system-level data real-time acquisition and downloading system based on HTTP streaming transmission, which comprises a request triggering module used for sending a data request through a preset interface; the server-side parameter verification module is used for performing parameter verification on the data request and outputting a verification result; the streaming output control module is used for outputting an HTTP response header after the verification is passed; the system data generation module is used for starting a system-level data generation program and transmitting the data through an HTTP (Hyper Text Transport Protocol) channel; the block transmission module is used for carrying out block processing on data needing to be transmitted and dividing the data into data blocks in a preset format; and the session termination control module is used for stopping data stream transmission according to a preset termination condition. Intermediate data files or caches are not needed, and occupation of system storage resources is remarkably reduced; the client does not need to cache complete data, and memory resource occupation is low; and real-time acquisition and long-term continuous output of system-level data are supported.

Description

Technical Field

[0001] This invention belongs to the field of data transmission technology, and in particular relates to a system-level real-time data acquisition and download system based on HTTP streaming transmission. Background Technology

[0002] In network devices, embedded systems, and maintenance / debugging scenarios, it is often necessary to remotely acquire data generated within the system, such as network packet capture data, system operation logs, and real-time diagnostic output information. Existing acquisition solutions typically include:

[0003] 1. File first, then download: Generate the complete file on the device's internal storage medium; then download it via HTTP / FTP or other methods.

[0004] 2. Data return scheme based on POST / API: Data collection is triggered via HTTPPOST or AJAX requests; the server returns the collected results as a response.

[0005] 3. WebSocket / Long Connection Solution: Maintain a long connection with the client via WebSocket and other methods; push system data in real time.

[0006] Existing acquisition methods suffer from large storage space consumption and high system resource consumption. Summary of the Invention

[0007] The purpose of this invention is to provide a system-level real-time data acquisition and download system based on HTTP streaming transmission, which aims to solve the problems of large storage space consumption and high system resource consumption in existing acquisition schemes.

[0008] This invention is implemented as follows: a system-level real-time data acquisition and download system based on HTTP streaming transmission, the system comprising:

[0009] A request triggering module, which is used to send a data request through a preset interface;

[0010] A server-side parameter validation module is used to validate the parameters of data requests and output the validation results.

[0011] A streaming output control module, which is used to output an HTTP response header after successful verification;

[0012] The system data generation module is used to start a system-level data generation program and transmit data through an HTTP channel.

[0013] The block transmission module is used to process the data to be transmitted into blocks, dividing it into data blocks of a preset format.

[0014] A session termination control module is used to stop data stream transmission according to preset termination conditions.

[0015] Preferably, the data request includes at least the data source and the data collection time or termination condition.

[0016] Preferably, during the process of performing parameter validation on the data request and outputting the validation results, the validation includes valid character validation, data source validity validation, and maximum collection duration validation.

[0017] Preferably, the response header includes at least Content-Disposition:attachment; disabling Content-Length; and enabling Transfer-Encoding:chunked.

[0018] Preferably, the data block includes at least a data block length field, data block content, and a delimiter.

[0019] Preferably, the termination conditions include the collection time reaching a preset value and the system process ending abnormally.

[0020] Preferably, the system-level data generation program is a packet capture program.

[0021] Preferably, data is transmitted synchronously via HTTP while the system-level data generation program generates data.

[0022] Preferably, if the verification fails, an error response is returned and the data stream output stage is not entered.

[0023] The system-level real-time data acquisition and download system based on HTTP streaming provided by this invention eliminates the need for intermediate data files or caches, significantly reducing system storage resource consumption; the client does not need to cache complete data, resulting in low memory resource consumption; it supports real-time acquisition and long-term continuous output of system-level data; it is compatible with existing HTTP protocols and browser native download mechanisms; and it is particularly suitable for resource-constrained embedded system environments. Attached Figure Description

[0024] Figure 1 This is an architecture diagram of a system-level real-time data acquisition and download system based on HTTP streaming transmission, provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0026] In existing technologies, methods for acquiring system-level data mainly include generating files locally and then downloading them over the network, returning complete response data via an HTTP interface, or pushing data via long-connection protocols such as WebSocket. However, the above solutions generally suffer from the following drawbacks:

[0027] First, it requires a large amount of local storage space, making it unsuitable for embedded devices with limited storage resources;

[0028] Secondly, the client needs to cache the complete response data, which leads to excessive memory resource consumption;

[0029] Third, it is difficult to support long-term or continuous data output, and its stability is poor.

[0030] The purpose of this invention is to provide a method that eliminates the need to generate intermediate files and requires no client-side caching of complete data.

[0031] This invention relates to a system-level method and apparatus for real-time data acquisition and download that supports long-term or continuous operation and is compatible with existing HTTP protocols and browser download mechanisms, addressing the issues of high resource consumption and poor real-time performance in existing technologies. The core idea is to directly map the real-time data stream generated internally into an HTTP response body and use a chunked transfer encoding mechanism for continuous output, thereby enabling the client to persistently download data without overall caching.

[0032] like Figure 1 The diagram shows a flowchart of a system-level real-time data acquisition and download system based on HTTP streaming provided in an embodiment of the present invention. The system includes:

[0033] A request triggering module is used to send data requests through a preset interface.

[0034] In this system, clients access preset interfaces via HTTP GET requests, for example:

[0035] https: / / m.home / cgi-bin / http.cgi? ifname=eth0&duration=3600

[0036] The request includes the following parameters: ① the data source to be collected (such as the network interface name); ② the collection duration or termination conditions.

[0037] The server-side parameter verification module is used to verify the parameters of data requests and output the verification results.

[0038] In this system, the request parameters are validated as follows before the response begins: valid character verification to prevent injection risks; data source validity verification; and maximum collection duration verification. If the verification fails, an error response is returned directly, and the data streaming output stage is not initiated.

[0039] A streaming output control module is used to output an HTTP response header after successful verification.

[0040] In this system, after successful parameter validation, the server outputs an HTTP response header all at once. The response header includes at least: Content-Disposition:attachment; disabling Content-Length; enabling Transfer-Encoding:chunked; thereby forcing the client to enter streaming download mode.

[0041] The system data generation module is used to start a system-level data generation program and transmit data through an HTTP channel.

[0042] In this system, the server starts a system-level data generation program (such as a packet capture process) and redirects its standard output to the HTTP output channel. The data generation process is synchronized with the HTTP transmission process and does not rely on intermediate storage.

[0043] The block transmission module is used to divide the data to be transmitted into blocks of a preset format.

[0044] In this system, the server divides the data obtained from the system data source into multiple data blocks and outputs them one by one according to the HTTP chunked transfer format, including: data block length field, data block content, and delimiter. By continuously outputting data blocks, the HTTP connection is maintained until the termination condition is met or the client actively disconnects.

[0045] A session termination control module is used to stop data stream transmission according to preset termination conditions.

[0046] In this system, the data stream will be terminated if any of the following conditions are met: the preset collection time is reached; the client disconnects; or the system process terminates abnormally. Upon termination, the server closes the data source and ends the HTTP session.

[0047] This invention combines the system-level data generation process with HTTP download semantics to construct a cache-free, low-resource-consumption real-time data acquisition path, effectively overcoming the technical problem that traditional system-level data acquisition schemes are difficult to run stably for a long time in embedded environments.

[0048] This invention does not simply transmit existing data in chunks, but rather directly triggers the system-level data generation process through the download semantics of the HTTP GET request. This ensures that the data being transmitted has not yet been generated and its length is unpredictable when the request is initiated, thus distinguishing it from existing HTTP chunked transmission methods that are only used for transmitting existing or predictable data.

[0049] This invention outputs an HTTP response header indicating download behavior once before system-level data is generated, and disables the content length field and enables chunked transmission mechanism, so that the client enters streaming download and persistent storage mode before receiving any system data. This uses the order of response header output to strongly constrain client behavior, which is different from the existing application of HTTP chunked transmission as only a means of data transmission.

[0050] This invention constructs a mapping transmission path for system-level data without intermediate storage, enabling system-level data to be directly sent to the client via HTTP chunked transmission after the data is generated. No intermediate files or complete caches are formed during the entire generation and transmission process, which solves the problem of long-term storage and caching of system data in resource-constrained devices, and is different from existing chunked transmission technologies that rely on application-layer caching.

[0051] In a specific embodiment of the present invention:

[0052] The detailed flowchart is as follows:

[0053] S101, The client initiates an HTTP GET request (carrying system-level data retrieval parameters);

[0054] S102. The server receives the request and parses the query parameters;

[0055] S103. Perform validity checks on the query parameters (interface whitelist, duration threshold); determine if the parameters are valid? If valid, proceed to S104; if invalid, return an error response.

[0056] S104. Output HTTP response headers (including attachment, chunked enabled, and no Content-Length);

[0057] S105. Start the system-level data generation process (network packet capture / log collection);

[0058] S106: Read system-level data and generate process output;

[0059] S107. Continuously send data blocks according to the HTTP chunked transfer format (no intermediate files, executed in a loop);

[0060] S108. Determine whether the termination condition is met (including duration reached / client disconnected / process terminated); if the termination condition is met, proceed to S109; if the termination condition is not met, return to S106.

[0061] S109. Close the data generation process and terminate the HTTP connection;

[0062] S110: The client receives data and writes it to the persistent storage medium in real time.

[0063] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0065] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A system-level real-time data acquisition and download system based on HTTP streaming transmission, characterized in that, The system includes: A request triggering module, which is used to send a data request through a preset interface; A server-side parameter validation module is used to validate the parameters of data requests and output the validation results. A streaming output control module, which is used to output an HTTP response header after successful verification; The system data generation module is used to start a system-level data generation program and transmit data through an HTTP channel. The block transmission module is used to process the data to be transmitted into blocks, dividing it into data blocks of a preset format. A session termination control module is used to stop data stream transmission according to preset termination conditions.

2. The system-level real-time data acquisition and download system based on HTTP streaming transmission according to claim 1, characterized in that, The data request includes at least the data source and the data collection time or termination condition.

3. The system-level real-time data acquisition and download system based on HTTP streaming transmission according to claim 1, characterized in that, The process of validating parameters for data requests and outputting the validation results includes valid character verification, data source validity verification, and maximum collection duration verification.

4. The system-level real-time data acquisition and download system based on HTTP streaming transmission according to claim 1, characterized in that, The response header must include at least Content-Disposition:attachment; disable Content-Length; enable Transfer-Encoding:chunked.

5. The system-level real-time data acquisition and download system based on HTTP streaming transmission according to claim 1, characterized in that, A data block must include at least a data block length field, data block content, and a delimiter.

6. The system-level real-time data acquisition and download system based on HTTP streaming transmission according to claim 1, characterized in that, Termination conditions include the collection time reaching a preset value and the system process ending abnormally.

7. The system-level real-time data acquisition and download system based on HTTP streaming transmission according to claim 1, characterized in that, System-level data generation program and packet capture program.

8. The system-level real-time data acquisition and download system based on HTTP streaming transmission according to claim 1, characterized in that, Data is transmitted synchronously via HTTP while the system-level data generation program generates data.

9. The system-level real-time data acquisition and download system based on HTTP streaming transmission according to claim 3, characterized in that, If the verification fails, an error response is returned and the data stream output stage is not entered.

Images