A multi-channel asynchronous distribution processing method and system based on a ZIP binary stream

Abstract

The application relates to a multi-channel asynchronous distribution processing system based on a ZIP binary stream and a multi-channel asynchronous distribution processing method based on a ZIP binary stream. The multi-channel asynchronous distribution processing system based on the ZIP binary stream comprises a stream processing module, a file processing module and a back pressure control and monitoring module. The stream processing module is used for receiving a ZIP binary stream and parsing the ZIP binary stream. The file processing module is used for distributing the parsed ZIP binary stream to multiple file processing channels which are independently arranged according to file types. The back pressure control and monitoring module is used for monitoring the queue state of the ZIP binary stream of the multiple file processing channels and adjusting the receiving speed of the stream processing module according to the monitored queue state. According to the application, resource occupation can be greatly reduced, processing speed can be significantly improved, and system stability and reliability can be increased.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and more specifically to a multi-channel asynchronous distribution processing method and system based on ZIP binary streams. Background Technology

[0002] To enrich the knowledge slice management of the intelligent question-answering platform, in addition to the platform's own knowledge segmentation capabilities, it is planned to synchronize external slice information to the platform as unstructured data for the intelligent question-answering platform. The knowledge slice synchronization involves diverse knowledge files, and the synchronization will be performed based on binary streams. Files will be distinguished by naming conventions to achieve automatic identification, classification, and distribution to unstructured databases, structured databases, and file storage.

[0003] In the context of knowledge slice synchronization, the source data not only contains massive amounts of unstructured text and multimedia slices, but also structured metadata configuration files, and the entire carrier ZIP package can reach a maximum size of 5GB. Traditional streaming distribution and batch processing technologies face the following five contradictions when dealing with such extreme loads: (1) Lack of real-time classification and distribution capability for multiple types of files in ZIP packages: Existing technologies have not achieved pipeline processing of "receiving, decompressing, classifying and distributing at the same time" during the ZIP binary stream reception process. Different types of files (original resource files, metadata, knowledge slices, etc.) in ZIP packages are treated indiscriminately and cannot enter the optimal processing path according to business needs. (2) High memory usage: Some existing technologies adopt a multi-stage mode of "complete reception--middleware caching--complete decompression--read each file--process separately", which generates a large amount of redundant disk I / O (i.e. input / output). The compressed package and the full decompressed data need to reside in the memory at the same time, resulting in high peak memory usage. (3) Serial processing leads to low overall efficiency: Existing technologies mostly process files in ZIP packages in a serial mode (processing files sequentially), which cannot take advantage of the parallelism between processing tasks of different types of files. When the ZIP package contains a large number of small files, the cumulative latency of serial processing increases significantly. (4) Lack of parallel distribution mechanism for multi-target systems: Existing technologies usually direct the processing results to a single target system (such as only writing to the database or only transmitting), without designing a mechanism to simultaneously distribute to multiple heterogeneous target systems such as NFS (file storage), database (metadata storage), and HTTP server (knowledge slice upload), which cannot meet the diverse processing needs in complex business scenarios. (5) Lack of dynamic back pressure and resource self-adaptation capability: Existing technologies lack effective back pressure control mechanisms when processing speeds are mismatched (such as network jitter causing HTTP upload delays or excessive instantaneous pressure on the database), which can easily lead to unlimited memory growth or system overload and crash. Summary of the Invention

[0004] In order to solve the problems in the prior art, the present invention aims to provide a multi-channel asynchronous distribution processing method and a multi-channel asynchronous distribution processing system based on ZIP binary streams that can reduce resource consumption and improve processing speed.

[0005] A multi-channel asynchronous distribution processing system based on ZIP binary streams according to one aspect of the present invention includes: A stream processing module is used to receive and parse a ZIP binary stream. The file processing module is used to distribute the parsed ZIP binary stream to multiple independently configured file processing channels according to file type; and The back pressure control and monitoring module is used to monitor the queue status of the ZIP binary streams of the multiple file processing channels and adjust the receiving rate of the stream processing module to receive the ZIP binary streams based on the monitored queue status.

[0006] Optionally, the stream processing module includes: A ZIP binary stream receiving module for receiving ZIP binary streams; and A streaming parser for single-pass streaming parsing of ZIP binary streams.

[0007] Optionally, the file processing module includes: File processors used to distribute the parsed ZIP to the multiple file processing channels; and Multiple file processing channels are used to handle different file types separately.

[0008] Optionally, the file processor reads the metadata of each file entry in the ZIP binary stream one by one, determines the file processing channel of the file type to which the file entry belongs according to the preset file classification rules based on the metadata of each file entry, and distributes the file entry to the corresponding file processing channel.

[0009] Optionally, the plurality of file processing channels include: The network file system write channel is used to asynchronously write the parsed ZIP binary stream to the target path; The database write channel is used to asynchronously store the parsed ZIP binary stream into the database; The HTTP upload channel is used to asynchronously upload knowledge slice files from the parsed ZIP binary stream via an HTTP interface.

[0010] Optionally, the network file system write channel is configured with a memory buffer of a specified size and the parsed ZIP binary stream is directly written to the target path of the remote network file system, and the memory buffer is released immediately upon completion of processing.

[0011] Optionally, the database write channel is used to parse each line of a structured file into a structured object immediately after reading it, and assemble it into a database record along with the corresponding network file system file path in the network file system write channel and write it into a data table.

[0012] Optionally, the HTTP upload channel reads the contents of the files to be uploaded one by one and then sends a POST request to the backend server through an HTTP client.

[0013] Optionally, the back pressure control and monitoring module monitors the length of the pending queue of the ZIP binary stream of the multiple file processing channels and pauses or slows down the receiving rate of the ZIP binary stream of the stream processing module when the length of the pending queue exceeds a preset threshold.

[0014] Optionally, the back pressure control and monitoring module is further used to monitor the memory usage of the multi-channel asynchronous distribution processing system and reduce the number of concurrent tasks of the multiple file processing channels when the memory usage exceeds a preset safety threshold.

[0015] Optionally, the preset file classification rules include one or more of the following: file extension, directory, and file size threshold.

[0016] The multi-channel asynchronous distribution processing method based on ZIP binary streams according to one aspect of the present invention includes the following steps: Receives a ZIP binary stream and is used to parse the ZIP binary stream; The parsed ZIP binary stream is distributed to multiple independently configured file processing channels according to file type; Monitor the queue status of the ZIP binary streams of the multiple file processing channels described above; and The receiving rate of the ZIP binary stream by the stream processing module is adjusted according to the monitored queue status.

[0017] Optionally, the ZIP binary stream can be parsed using a single-pass streaming method.

[0018] Optionally, distributing the parsed ZIP binary stream to multiple independently configured file processing channels according to file type includes: Read the metadata of each file entry in the ZIP binary stream, entry by entry; Based on the metadata of each read file entry, the file processing channel determines the file type to which the file entry belongs according to preset file classification rules; and Distribute the file entry to the corresponding file processing channel.

[0019] Optionally, the plurality of file processing channels include: The network file system write channel is used to asynchronously write the parsed ZIP binary stream to the target path; The database write channel is used to asynchronously store the parsed ZIP binary stream into the database; The HTTP upload channel is used to asynchronously upload knowledge slice files from the parsed ZIP binary stream via an HTTP interface.

[0020] Optionally, the network file system write channel is configured with a memory buffer of a specified size and the parsed ZIP binary stream is directly written to the target path of the remote network file system, and the memory buffer is released immediately upon completion of processing.

[0021] Optionally, the database write channel is used to parse each line of a structured file into a structured object immediately after reading it, and assemble it into a database record along with the corresponding network file system file path in the network file system write channel and write it into a data table.

[0022] Optionally, the HTTP upload channel reads the contents of the files to be uploaded one by one and then sends a POST request to the backend server through an HTTP client.

[0023] Optionally, monitoring the queue status of the ZIP binary streams of the multiple file processing channels and adjusting the receiving rate of the stream processing module to receive the ZIP binary streams based on the monitored queue status includes: The system monitors the length of the pending queue of the ZIP binary streams of the multiple file processing channels and pauses or slows down the receiving rate of the ZIP binary stream of the stream processing module when the length of the pending queue exceeds a preset threshold.

[0024] Optionally, monitoring the queue status of the ZIP binary streams of the multiple file processing channels and adjusting the receiving rate of the stream processing module to receive the ZIP binary streams based on the monitored queue status further includes: Monitor system memory usage and reduce the number of concurrent tasks in the multiple file processing channels when the system memory usage exceeds a preset safety threshold.

[0025] Optionally, the preset file classification rules include one or more of the following: file extension, directory, and file size threshold.

[0026] A computer-readable medium of the present invention stores a computer program thereon, which, when executed by a processor, implements the multi-channel asynchronous distribution processing method described above.

[0027] A computer device according to one aspect of the present invention includes a storage module, a processor, and a computer program stored on the storage module and executable on the processor, wherein the processor executes the computer program to implement the intent penetration testing method.

[0028] A computer program product according to one aspect of the present invention includes a computer program that, when executed by a processor, implements the aforementioned intent penetration testing method. Attached Figure Description

[0029] The described and other objects and advantages of the invention will become more fully clear from the following detailed description taken in conjunction with the accompanying drawings, wherein like or similar elements are denoted by the same reference numerals.

[0030] Figure 1 This is a block diagram illustrating the structure of a multi-channel asynchronous distribution processing system based on ZIP binary streams according to an embodiment of the present invention.

[0031] Figure 2 This is a flowchart illustrating a multi-channel asynchronous distribution processing method based on ZIP binary streams according to an embodiment of the present invention.

[0032] Figure 3 This is a flowchart illustrating an example of back pressure control and resource monitoring. Detailed Implementation

[0033] The following are some embodiments of the present invention, intended to provide a basic understanding of the invention. They are not intended to identify key or decisive elements of the invention or to limit the scope of protection sought.

[0034] For purposes of brevity and illustrativeness, the principles of the invention are described herein primarily with reference to exemplary embodiments thereof. However, those skilled in the art will readily recognize that the same principles are equivalently applicable to all types of multichannel asynchronous distribution processing methods and systems based on ZIP binary streams, and that these same principles can be implemented therein, with any such variations not departing from the true spirit and scope of the invention.

[0035] Furthermore, reference is made in the following description to the accompanying drawings, which illustrate specific exemplary embodiments. Electrical, mechanical, logical, and structural modifications may be made to these embodiments without departing from the spirit and scope of the invention. Moreover, while features of the invention are disclosed in conjunction with only one of several embodiments, this feature may be combined with one or more other features of other embodiments if desired and / or advantageous for any given or identifiable function. Therefore, the following description should not be considered limiting in any sense, and the scope of the invention is defined by the appended claims and their equivalents.

[0036] Terms such as “possessing” and “comprising” indicate that, in addition to having units (modules) and steps that are directly and explicitly stated in the specification and claims, the technical solution of the present invention does not exclude the presence of other units (modules) and steps that are not directly or explicitly stated.

[0037] A multi-channel asynchronous distribution processing system based on ZIP binary streams according to one aspect of the present invention includes: A stream processing module is used to receive and parse a ZIP binary stream. The file processing module is used to distribute the parsed ZIP binary stream to multiple independently configured file processing channels according to file type; and The back pressure control and monitoring module is used to monitor the queue status of the ZIP binary streams of the multiple file processing channels and adjust the receiving rate of the stream processing module to receive the ZIP binary streams based on the monitored queue status.

[0038] Therefore, the stream processing module is responsible for receiving and parsing ZIP streams, the file processing module implements multi-type file splitting and processing, and the back pressure control and monitoring module manages the upstream and downstream rates in real time. This solves the shortcomings of traditional solutions, such as the lack of unified traffic control and mixed file processing, while supporting pipeline operations that receive and process simultaneously.

[0039] Optionally, the stream processing module includes: a ZIP binary stream receiving module for receiving ZIP binary streams; and a stream parser for single-pass streaming parsing of ZIP binary streams.

[0040] Therefore, by implementing single-pass streaming parsing based on the streaming parser, there is no need to receive and process the complete ZIP package on disk, thus eliminating the high memory consumption caused by caching the complete compressed package. At the same time, it saves multiple rounds of disk read and write, greatly reducing redundant I / O, and the peak memory usage can be reduced by more than 80%.

[0041] Optionally, the file processing module includes: a file processor for distributing the parsed ZIP to the plurality of file processing channels; and a plurality of file processing channels for processing different file types respectively.

[0042] In this way, the file processor uniformly completes the file classification and distribution logic, and with multiple independent file processing channels, it achieves business isolation. This breaks the limitation of traditional solutions that can only output data to a single target system. It can simultaneously connect to heterogeneous systems such as file storage, databases, and remote services, meeting the complex business needs of multi-destination synchronization of knowledge slices.

[0043] Optionally, the file processor reads the metadata of each file entry in the ZIP binary stream one by one, determines the file processing channel of the file type to which the file entry belongs according to the preset file classification rules based on the metadata of each file entry, and distributes the file entry to the corresponding file processing channel.

[0044] Therefore, by reading file metadata item by item and allocating channels in real time according to rules, a pipelined processing of decompression, classification, and distribution during ZIP stream reception is achieved. This eliminates the need for indiscriminate processing of all files within the compressed package; each file type can directly enter its optimal dedicated processing path, shortening service response latency. Optionally, the multiple file processing channels include: a network file system write channel for asynchronously writing the parsed ZIP binary stream to the target path; a database write channel for asynchronously storing the parsed ZIP binary stream into a database; and an HTTP upload channel for asynchronously uploading knowledge slice files from the parsed ZIP binary stream via an HTTP interface.

[0045] By setting up different file processing channels to correspond to resource storage, metadata entry, knowledge slice upload, etc., the multi-channel asynchronous parallel execution can eliminate the cumulative latency caused by traditional serial processing of a large number of small files, and the overall processing speed is 3 to 5 times faster than the serial solution.

[0046] Optionally, the network file system write channel is configured with a memory buffer of a specified size and the parsed ZIP binary stream is directly written to the target path of the remote network file system, and the memory buffer is released immediately upon completion of processing.

[0047] Therefore, the network file system write channel adopts a fixed memory buffer to write directly to the remote storage, without generating local temporary disk files throughout the process. The buffer is released immediately after processing is completed, and the zero-disk direct memory transfer architecture completely eliminates the double redundant disk I / O caused by local disk transfer, reducing disk load and memory usage.

[0048] Optionally, the database write channel is used to parse each line of a structured file into a structured object immediately after reading it, and assemble it into a database record along with the corresponding network file system file path in the network file system write channel and write it into a data table.

[0049] Therefore, the database write channel parses structured data line by line, releasing memory as soon as a single line is processed, avoiding the large amount of memory occupied by loading the entire structured file at once. At the same time, the NFS storage path associated with the network file system write channel is uniformly stored in the database, ensuring that the metadata and resource file storage information correspond one-to-one, resulting in stronger data integrity.

[0050] Optionally, the HTTP upload channel reads the contents of the files to be uploaded one by one and then sends a POST request to the backend server through an HTTP client.

[0051] Therefore, the HTTP upload channel sends knowledge slice POST requests one by one, supports independent retries, timeouts, and connection pooling mechanisms. If the channel fails, it will only retry internally and will not block other business processes such as NFS writes and database insertions, thus achieving fault isolation and improving the overall stability of the system.

[0052] Optionally, the back pressure control and monitoring module monitors the length of the pending queue of the ZIP binary stream of the multiple file processing channels and pauses or slows down the receiving rate of the ZIP binary stream of the stream processing module when the length of the pending queue exceeds a preset threshold.

[0053] Therefore, by monitoring the length of the queue to be processed in each channel in real time, the ZIP stream reception is reduced or suspended if the queue backlog exceeds the limit. This controls the data inflow speed from the upstream source, solves the problem of continuous data accumulation and infinite memory expansion caused by the mismatch between upstream and downstream processing rates, and avoids system memory overflow and crash.

[0054] Optionally, the back pressure control and monitoring module is further used to monitor the system memory usage and reduce the number of concurrent tasks of the multiple file processing channels when the system memory usage exceeds a preset safety threshold.

[0055] Therefore, the back pressure control and monitoring module, through its memory monitoring capabilities, automatically reduces the number of concurrent tasks in all channels when memory usage exceeds the limit, thereby achieving adaptive regulation of system resources, balancing the consumption of CPU and memory resources of the entire machine, preventing a single channel from monopolizing all hardware resources, and ensuring the long-term stable operation of the system under high load.

[0056] Optionally, the preset file classification rules include one or more of the following: file extension, directory, and file size threshold.

[0057] The multi-channel asynchronous distribution processing method based on ZIP binary streams according to one aspect of the present invention includes the following steps: Receives a ZIP binary stream and is used to parse the ZIP binary stream; The parsed ZIP binary stream is distributed to multiple independently configured file processing channels according to file type; and The system monitors the queue status of the ZIP binary streams of the multiple file processing channels and adjusts the receiving rate of the ZIP binary streams received by the stream processing module based on the monitored queue status.

[0058] Optionally, the ZIP binary stream can be parsed using a single-pass streaming method.

[0059] Optionally, distributing the parsed ZIP binary stream to multiple independently configured file processing channels according to file type includes: Read the metadata of each file entry in the ZIP binary stream, entry by entry; Based on the metadata of each read file entry, the file processing channel determines the file type to which the file entry belongs according to preset file classification rules; and Distribute the file entry to the corresponding file processing channel.

[0060] Optionally, the plurality of file processing channels include: The network file system write channel is used to asynchronously write the parsed ZIP binary stream to the target path; The database write channel is used to asynchronously store the parsed ZIP binary stream into the database; The HTTP upload channel is used to asynchronously upload knowledge slice files from the parsed ZIP binary stream via an HTTP interface.

[0061] Optionally, the network file system write channel is configured with a memory buffer of a specified size and the parsed ZIP binary stream is directly written to the target path of the remote network file system, and the memory buffer is released immediately upon completion of processing.

[0062] Optionally, the database write channel is used to parse each line of a structured file into a structured object immediately after reading it, and assemble it into a database record along with the corresponding network file system file path in the network file system write channel and write it into a data table.

[0063] Optionally, the HTTP upload channel reads the contents of the files to be uploaded one by one and then sends a POST request to the backend server through an HTTP client.

[0064] Optionally, monitoring the queue status of the ZIP binary streams of the multiple file processing channels and adjusting the receiving rate of the stream processing module to receive the ZIP binary streams based on the monitored queue status includes: The system monitors the length of the pending queue of the ZIP binary streams of the multiple file processing channels and pauses or slows down the receiving rate of the ZIP binary stream of the stream processing module when the length of the pending queue exceeds a preset threshold.

[0065] Optionally, monitoring the queue status of the ZIP binary streams of the multiple file processing channels and adjusting the receiving rate of the stream processing module to receive the ZIP binary streams based on the monitored queue status further includes: Monitor system memory usage and reduce the number of concurrent tasks in the multiple file processing channels when the system memory usage exceeds a preset safety threshold.

[0066] Optionally, the preset file classification rules include one or more of the following: file extension, directory, and file size threshold.

[0067] Next, a multi-channel asynchronous distribution processing system based on ZIP binary streams according to a specific embodiment of the present invention will be described.

[0068] Figure 1 This is a block diagram illustrating the structure of a multi-channel asynchronous distribution processing system based on ZIP binary streams according to an embodiment of the present invention.

[0069] like Figure 1 As shown, a multi-channel asynchronous distribution processing system based on ZIP binary streams, representing an embodiment of the present invention, includes: The stream processing module 100 includes: a ZIP binary stream receiving module 110 for receiving ZIP binary streams and a stream parser 120 for parsing ZIP binary streams; File processing module 200 includes: a file processor 210 for distributing the parsed ZIP to various processing channels, and one or more file processing channels, wherein... Figure 1 The document lists NFS write channel 220 (NFS, Network File System), database write channel 230, HTTP upload channel 240, and other channels 250); and The back pressure control and monitoring module 300 includes: a monitoring module 310 for monitoring the queue threshold of each file processing channel and a flow rate regulator 320 for adjusting the receiving rate according to the monitoring results.

[0070] Figure 2This is a flowchart illustrating a multi-channel asynchronous distribution processing method based on ZIP binary streams according to an embodiment of the present invention.

[0071] like Figure 2 As shown, a multi-channel asynchronous distribution processing method based on ZIP binary stream according to an embodiment of the present invention includes the following steps: Step S0: Start the process; Step S1: Receive ZIP binary stream, i.e., establish a ZIP binary stream receiving channel, which specifically includes: the client sending a request to the server and establishing a long connection, the server returning a binary stream response in ZIP format, and the client obtaining the input stream; Step S2: Initialize the streaming ZIP parser, which includes: the client creating a streaming ZIP parser instance to wrap the above input stream, and setting the parsing buffer size (e.g., 8KB) to prepare for item-by-item parsing; Step S3: Determine if there is a next item (i.e., start item-by-item streaming parsing and classification). If the result is "yes", continue to step S4. If the result is "no", jump to step S8. Step S4: Read entry metadata, that is, read each file entry in the ZIP package in a loop. Each entry includes: a. Read the metadata of the entry (filename, file size, compression method, modification time, etc.); b. Determine the processing channel type of the file based on preset classification rules (such as file extension, directory, file size threshold, etc.), for example, NFS write channel, database write channel, HTTP upload channel, other channels, etc. Step S5: Determine the file type, i.e., distribute the file to the corresponding processing channel according to its type, for example including: S5-1: Asynchronous write to NFS channel (i.e., NFS write channel): The file stream of the entry is directly written to the mounted target path through the NFS client. A fixed-size memory buffer (e.g., 64KB) is used. The ZIP binary stream is read in a loop and written to NFS. After writing, the file handle is closed and the buffer is released immediately. This operation is performed in a separate asynchronous task. S5-2: Asynchronous parsing and incorporation into the database (i.e., database write channel): For structured files, a line-by-line streaming approach is used. Each line read is immediately parsed into a structured object (such as file size, file name, upload time, file type, etc.), and together with the corresponding NFS storage path in S5-1, it is assembled into a database record and written to the data table through batch insertion or line-by-line insertion. This operation is performed in an independent asynchronous task. S5-3: Asynchronous HTTP Requests (and HTTP Upload Channel): For files that need to be uploaded, such as knowledge slices, after reading the content one by one, a POST request is sent to the backend server through an HTTP client. A connection pool and timeout mechanism are set up to support retry on failure. This operation is executed in an independent asynchronous task. S5-4: Other processing / skip: For files that do not conform to the above types, you can choose to skip, log, or enter the custom processing channel; Step S6: Close the current entry and process the next one: After the current entry is processed, close it, release the relevant resources, and continue to process the next entry in a loop until all entries are processed; Step S7: Back pressure control and resource monitoring, which includes continuously monitoring the buffer status and processing rate of each processing channel during the execution of steps 4 to S6, specifically including: a. When the length of the queue to be processed in a certain channel exceeds a preset threshold, the back pressure mechanism is triggered to pause or slow down the receiving rate of the ZIP binary stream; b. When system memory usage exceeds the safety threshold, reduce the number of concurrent tasks on each channel; c. When a channel fails to process, an error log is recorded, and the channel is retried or skipped according to the policy, without affecting the normal processing of other channels; Step S8: After all entries have been processed, close the parsing stream and summarize the processing results of each channel (number of successes, number of failures, processing time, etc.) to complete the entire ZIP packet processing flow. Step S9: End.

[0072] Next, refer to Figure 3 This section will illustrate the specific process of an example of back pressure control and resource monitoring.

[0073] Figure 3 This is a flowchart illustrating an example of back pressure control and resource monitoring.

[0074] like Figure 3 As shown, the cyclical process of back pressure control and resource monitoring includes: Step S7-0: Begin; Step S7-1: Calculate the processing queue length for each channel; Step S7-2: Monitor the status of each channel and continuously perform back pressure checks and resource adjustments; Step S7-3: Determine whether there is a channel processing queue length that exceeds the preset backlog threshold. If the determination result is "yes", continue to step S7-4. If the determination result is "no", jump to step S7-8. Step S7-4: Identify the bottleneck channel; Step S7-5: Reduce the ZIP binary stream receive rate or pause reception; Step S7-6: Determine whether the bottleneck channel has dropped to a safe water level. If the determination result is "yes", then continue to step S7-7. Step S7-7: Resume ZIP binary stream reception; Steps S7-8: Do not modify the ZIP binary stream receive rate, proceed normally.

[0075] Based on the above description, the multi-channel asynchronous distribution processing method based on ZIP binary streams and the system of the multi-channel asynchronous distribution processing method based on ZIP binary streams of the present invention adopt a single-pass streaming parsing and multi-channel real-time distribution mechanism. Specifically, single-pass streaming parsing means that without waiting for the complete transmission of the entire ZIP compressed package or storing the entire compressed package on the local disk, the client receives the ZIP binary data stream transmitted from the network and reads each file entry (i.e., Entry) inside the compressed package one by one. The entire ZIP stream is traversed and read only once, and only one round of parsing operation is performed throughout the entire process. There is no repeated reading of the compressed package, secondary decompression, or secondary file traversal (traditional solutions require receiving the complete package, writing it to disk, fully decompressing it, and then traversing the files, which requires at least 2 to 3 more rounds of I / O traversal).

[0076] The implementation logic relies on a streaming parser that continuously consumes network binary streams with a fixed small buffer (e.g., 8KB), extracting metadata (filename, extension, size, path) and file substreams for each file within the packet, without caching the entire compressed package to memory or disk. After parsing each file entry, the business type is immediately determined based on preset classification rules (file extension, storage directory, file size), without waiting for all files to be parsed. The corresponding substream for that file is immediately sent for processing, eliminating the latency associated with batch caching and then uniformly distributing all files. Furthermore, multiple independent asynchronous business processing channels are preset to cover heterogeneous storage and services. For example, an NFS write channel is used to store raw resource files, a database write channel is used to store file metadata and structured slice information, an HTTP upload channel is used to push knowledge slices to the backend server, and other custom channels are used to process files with special formats. In this way, different file streams are processed separately through their dedicated channels, without blocking each other.

[0077] Furthermore, the multi-channel asynchronous distribution processing method based on ZIP binary streams and the system based on ZIP binary streams of this invention adopt a zero-disk direct memory transfer architecture. "Zero-disk transfer" means that the complete file or compressed package is never written to the local disk, eliminating the intermediate step of "local disk transfer" in traditional technologies. All decompression and data processing logic is completed in the memory buffer, eliminating the additional read / write operations caused by disk transfer. For example, for files that need to be written to NFS, a direct write-to-NFS mount path is used, where the file stream is decompressed in memory and directly written to the target storage, avoiding redundant I / O of first writing to the local disk and then transferring. For structured data files, a line-by-line streaming read and real-time processing method is used, releasing memory immediately after each data entry is processed, strictly controlling peak memory usage. Thus, the dual read / write I / O of "local disk transfer" can be completely eliminated, reducing disk load.

[0078] Furthermore, the multi-channel asynchronous distribution processing method based on ZIP binary streams and the system based on the multi-channel asynchronous distribution processing method based on ZIP binary streams of this invention employ asynchronous non-blocking multi-channel parallel processing. Specifically, each processing channel executes in parallel using an asynchronous non-blocking mode, and the concurrent tasks of each channel are managed through coroutines or lightweight thread pools. When a certain channel processes slowly (such as due to network jitter causing HTTP upload delays), it will not affect the processing progress of other channels. Each channel independently maintains its processing status and retry mechanism, and each channel records its own task success / failure status separately, thus isolating faults. For example, if an HTTP upload fails, it will only be retried within the HTTP channel and will not interrupt NFS file storage or metadata entry; if a database write fails momentarily, only the database channel will retry, without affecting the services of other channels.

[0079] Furthermore, the multi-channel asynchronous distribution processing method based on ZIP binary streams and the system based on ZIP binary streams of the present invention employ dynamic backpressure and adaptive resource control. Buffer thresholds and processing rate monitoring are set for each channel. When the processing rate of a certain channel is lower than the receiving rate, a backpressure mechanism is automatically triggered to reduce the ZIP binary stream receiving speed or pause receiving, preventing unlimited memory growth. At the same time, the concurrency of each channel is dynamically adjusted according to the system load to achieve adaptive balance of resource usage.

[0080] Specifically, the system collects real-time data on the inflow rate of upstream ZIP streams, the backlog length of task queues in each channel, and the channel data processing rate. When a channel slows down (e.g., due to HTTP network lag or sudden database pressure), and pending tasks accumulate, exceeding a preset buffer threshold, channel congestion is identified. In this case, a rate-limiting command is automatically sent to the upstream ZIP stream receiving module. For example, in cases of mild congestion, the binary stream receiving speed is reduced; in cases of severe congestion, ZIP data stream reception is temporarily suspended. Furthermore, the bottleneck channel queue is continuously monitored, and normal ZIP stream receiving speed is restored once the backlog of data in the queue has been processed and the queue length has returned to a safe level. Compared to traditional solutions with no back pressure control, this invention's back pressure mechanism controls data inflow at the source, preventing unlimited memory expansion.

[0081] On the other hand, real-time monitoring of overall system resource metrics, such as global monitoring server CPU utilization, total memory usage, and system load data like task execution time for each channel, allows for dynamic adjustment of channel concurrency. For example, when the system is idle and resources are plentiful, the number of concurrent tasks on each channel is automatically increased to improve overall processing speed; when CPU / memory usage is near its limit, the number of concurrent tasks on all channels is automatically reduced to decrease resource consumption; or differentiated scheduling is implemented, with channels appropriately reducing concurrency while idle channels maintain high concurrency, achieving on-demand resource allocation. Thus, all channels share server hardware resources, preventing any single channel from monopolizing all computing power while other channels suffer from resource shortages, maintaining a balanced overall system resource utilization.

[0082] As described above, the multi-channel asynchronous distribution processing method based on ZIP binary stream and the system based on the multi-channel asynchronous distribution processing method based on ZIP binary stream according to the present invention can achieve the following technical effects: The zero-disk direct memory transfer architecture avoids the redundant disk I / O of "full disk write → decompression → reread" in the traditional solution. The memory usage is only the buffer size of the currently processed file, rather than the entire ZIP package size. The peak memory usage can be reduced by more than 80%, which greatly reduces resource consumption. The single-pass streaming parsing and multi-channel parallel processing mechanism eliminates the accumulated latency of serial processing. Each processing channel executes concurrently, and the overall processing time is close to the longest single-channel processing time, rather than the sum of the processing times of each channel. The processing speed is about 3 to 5 times faster than the traditional serial solution, which can significantly improve the processing speed. The dynamic backpressure mechanism prevents memory overflow, and the independent asynchronous processing and retry mechanism of each channel ensures that a single point of failure does not affect the overall process. The system has stronger abnormal recovery capabilities and high availability, which can enhance system stability and reliability. It does not require the introduction of additional components such as distributed caching and message middleware. It can achieve efficient distribution through local client-side streaming processing and lightweight concurrency control. It is simple to deploy, has low operation and maintenance costs, and can reduce the complexity of system architecture. Because of the use of streaming processing, various files in the ZIP package can enter their respective processing channels in real time during the receiving process. Metadata storage and knowledge slice uploading do not need to wait for all files to be processed, significantly reducing business response latency and improving business response real-time performance.

[0083] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art can conceive of other feasible variations or substitutions based on the technical scope disclosed in the present invention, and such variations or substitutions are all covered within the scope of protection of the present invention. In the absence of conflict, the embodiments of the present invention and the features thereof can also be combined with each other. The scope of protection of the present invention is determined by the claims.

Claims

1. A multi-channel asynchronous distribution processing system based on ZIP binary streams, characterized in that, include: A stream processing module is used to receive and parse a ZIP binary stream. The file processing module is used to distribute the parsed ZIP binary stream to multiple independently configured file processing channels according to file type. as well as The back pressure control and monitoring module is used to monitor the queue status of the ZIP binary streams of the multiple file processing channels and adjust the receiving rate of the stream processing module to receive the ZIP binary streams based on the monitored queue status.

2. The multi-channel asynchronous distribution processing system based on ZIP binary stream as described in claim 1, characterized in that, The stream processing module includes: A ZIP binary stream receiving module for receiving ZIP binary streams; and A streaming parser for single-pass streaming parsing of ZIP binary streams.

3. The multi-channel asynchronous distribution processing system based on ZIP binary stream as described in claim 1, characterized in that, The file processing module includes: File processors used to distribute the parsed ZIP to the multiple file processing channels; and Multiple file processing channels are used to handle different file types separately.

4. The multi-channel asynchronous distribution processing system based on ZIP binary stream as described in claim 3, characterized in that, The file processor reads the metadata of each file entry in the ZIP binary stream one by one, and determines the file processing channel of the file type to which the file entry belongs according to the preset file classification rules based on the metadata of each file entry, and distributes the file entry to the corresponding file processing channel.

5. The multi-channel asynchronous distribution processing system based on ZIP binary stream as described in claim 4, characterized in that, The multiple file processing channels include: The network file system write channel is used to asynchronously write the parsed ZIP binary stream to the target path; The database write channel is used to asynchronously store the parsed ZIP binary stream into the database; The HTTP upload channel is used to asynchronously upload knowledge slice files from the parsed ZIP binary stream via an HTTP interface.

6. The multi-channel asynchronous distribution processing system based on ZIP binary stream as described in claim 4, characterized in that, The network file system write channel is configured with a memory buffer of a specified size and writes the parsed ZIP binary stream directly to the target path of the remote network file system, releasing the memory buffer immediately upon completion of processing.

7. The multi-channel asynchronous distribution processing system based on ZIP binary stream as described in claim 4, characterized in that, The database write channel is used to parse each line of a structured file into a structured object immediately after reading it, and assemble it into a database record along with the corresponding file path of the network file system in the network file system write channel and write it into the data table.

8. The multi-channel asynchronous distribution processing system based on ZIP binary stream as described in claim 4, characterized in that, The HTTP upload channel reads the contents of the files to be uploaded one by one and then sends a POST request to the backend server through an HTTP client.

9. The multi-channel asynchronous distribution processing system based on ZIP binary stream as described in claim 1, characterized in that, The back pressure control and monitoring module monitors the length of the pending queue of the ZIP binary stream of the multiple file processing channels and pauses or slows down the receiving rate of the ZIP binary stream of the stream processing module when the length of the pending queue exceeds a preset threshold.

10. The multi-channel asynchronous distribution processing system based on ZIP binary stream as described in claim 1, characterized in that, The back pressure control and monitoring module is further used to monitor the memory usage of the multi-channel asynchronous distribution processing system and reduce the number of concurrent tasks of the multiple file processing channels when the memory usage exceeds a preset safety threshold.

11. The multi-channel asynchronous distribution processing system based on ZIP binary stream as described in claim 4, characterized in that, The preset file classification rules include one or more of the following: file extension, directory, and file size threshold.

12. A multi-channel asynchronous distribution processing method based on ZIP binary streams, characterized in that, Includes the following steps: Receives a ZIP binary stream and is used to parse the ZIP binary stream; The parsed ZIP binary stream is distributed to multiple independently configured file processing channels according to file type; Monitor the queue status of the ZIP binary streams of the multiple file processing channels described above; as well as The receiving rate of the ZIP binary stream by the stream processing module is adjusted according to the monitored queue status.

13. The multi-channel asynchronous distribution processing method based on ZIP binary stream as described in claim 12, characterized in that, The ZIP binary stream is parsed using a single-pass streaming method.

14. The multi-channel asynchronous distribution processing method based on ZIP binary stream as described in claim 12, characterized in that, The step of distributing the parsed ZIP binary stream to multiple independently configured file processing channels according to file type includes: Read the metadata of each file entry in the ZIP binary stream, entry by entry; Based on the metadata of each read file entry, the file processing channel determines the file type to which the file entry belongs according to preset file classification rules; and Distribute the file entry to the corresponding file processing channel.

15. The multi-channel asynchronous distribution processing method based on ZIP binary stream as described in claim 14, characterized in that, The multiple file processing channels include: The network file system write channel is used to asynchronously write the parsed ZIP binary stream to the target path; The database write channel is used to asynchronously store the parsed ZIP binary stream into the database; The HTTP upload channel is used to asynchronously upload knowledge slice files from the parsed ZIP binary stream via an HTTP interface.

16. The multi-channel asynchronous distribution processing method based on ZIP binary stream as described in claim 15, characterized in that, The network file system write channel is configured with a memory buffer of a specified size and writes the parsed ZIP binary stream directly to the target path of the remote network file system, releasing the memory buffer immediately upon completion of processing.

17. The multi-channel asynchronous distribution processing method based on ZIP binary stream as described in claim 15, characterized in that, The database write channel is used to parse each line of a structured file into a structured object immediately after reading it, and assemble it into a database record along with the corresponding file path of the network file system in the network file system write channel and write it into the data table.

18. The multi-channel asynchronous distribution processing method based on ZIP binary stream as described in claim 15, characterized in that, The HTTP upload channel reads the contents of the files to be uploaded one by one and then sends a POST request to the backend server through an HTTP client.

19. The multi-channel asynchronous distribution processing method based on ZIP binary stream as described in claim 12, characterized in that, The monitoring of the queue status of the ZIP binary streams of the multiple file processing channels and the adjustment of the receiving rate of the stream processing module to receive the ZIP binary streams based on the monitored queue status include: The system monitors the length of the pending queue of the ZIP binary streams of the multiple file processing channels and pauses or slows down the receiving rate of the ZIP binary stream of the stream processing module when the length of the pending queue exceeds a preset threshold.

20. The multi-channel asynchronous distribution processing method based on ZIP binary stream as described in claim 12, characterized in that, The monitoring of the queue status of the ZIP binary streams of the multiple file processing channels and the adjustment of the receiving rate of the stream processing module to receive the ZIP binary streams based on the monitored queue status further includes: Monitor system memory usage and reduce the number of concurrent tasks in the multiple file processing channels when the system memory usage exceeds a preset safety threshold.

21. The multi-channel asynchronous distribution processing method based on ZIP binary stream as described in claim 12, characterized in that, The preset file classification rules include one or more of the following: file extension, directory, and file size threshold.

22. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-channel asynchronous distribution processing method according to any one of claims 12 to 21.

23. A computer device, comprising a storage module, a processor, and a computer program stored on the storage module and executable on the processor, characterized in that, When the processor executes the computer program, it implements the intent penetration testing method according to any one of claims 12 to 21.

24. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the intent penetration testing method as described in any one of claims 12 to 21.

Images