A thread pool and queue-based container log asynchronous streaming method and system

Abstract

The application belongs to the technical field of containers, and discloses a container log asynchronous streaming method and system based on a thread pool and a queue, which comprises the following steps: receiving and analyzing a hypertext transfer protocol request parameter for obtaining target container logs initiated by a client, and processing the analyzed hypertext transfer protocol request parameter to generate a log production task; based on a pre-configured thread pool, receiving the log production task, obtaining a log stream of the log production task through a container engine and pre-processing the log stream, and transmitting the pre-processed log stream to a thread-safe queue for storage; transmitting the thread-safe queue to a pre-configured hypertext transfer protocol response header in a block encoding format, refreshing to the client, and judging whether the hypertext transfer protocol response header detects a termination signal or a thread event signal. The application can timely and completely release all resources that are no longer needed due to disconnection of the client, avoids resource leakage, and improves system stability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of containers, in particular to a container log asynchronous streaming method and system based on a thread pool and a queue. BACKGROUND

[0002] With the wide application of containerization technology (such as Docker), in cloud native platforms, continuous integration / continuous deployment (CI / CD) pipelines, and distributed simulation computing (such as unmanned aerial vehicle cluster simulation), it is often necessary to dynamically create and manage a large number of containers to perform tasks. Real-time acquisition of the running logs of these containers is crucial for task status monitoring, troubleshooting, and result analysis. In particular, in industrial cloud platforms and industrial internet platforms, containerization technology is further popularized. Such platforms often deploy industrial data acquisition services, device remote monitoring programs, production process scheduling modules, and other core industrial applications through containers, and need to manage hundreds or thousands of container instances corresponding to industrial devices. The logs generated by these containers not only contain the running status of the applications, but also associate with key information such as industrial device parameters and process execution data. Real-time and complete acquisition of logs is the core support for realizing real-time monitoring of production processes, rapid positioning of industrial device faults, and production safety compliance tracing. The low latency, high stability, and reasonable resource occupation of log transmission directly affect the continuous operation capability and industrial scenario service reliability of industrial cloud platforms / industrial internet platforms.

[0003] Traditional container log acquisition methods are usually synchronous and blocking. When a client requests logs, the server directly calls the log interface of the container engine, loads all the acquired log data into memory at once, and then returns the complete log data to the client. Therefore, the traditional method has the following significant shortcomings:

[0004] 1) High latency and memory pressure: For containers that run for a long time or have a large amount of logs, synchronous acquisition of all logs will produce significant latency, and the server process needs to consume a large amount of memory to store the complete log data, which risks running out of memory.

[0005] 2) Resource waste: If the client actively interrupts the connection during transmission (for example, the user closes the browser), the server may not be able to perceive it in time, causing the acquired log data to be discarded, while the production log thread or process continues to work, resulting in waste of computing and I / O resources.

[0006] 3) Poor scalability: The synchronous blocking model is difficult to handle high concurrency requests. Each log request may occupy a server worker process (or thread) for a long time, causing the server throughput to decrease and being unable to effectively serve more clients.

[0007] 4) Limited real-time: It is difficult to achieve a truly "follow" log function, that is, to continuously output new logs generated by the container. Even if this function is provided, its connection management and resource cleanup mechanism is often not robust enough.

[0008] Therefore, how to provide a container log asynchronous streaming method and system based on thread pool and queue is a problem to be solved at present. SUMMARY

[0009] The embodiment of the present application provides a container log asynchronous streaming method and system based on thread pool and queue, to solve the above technical problems in the prior art.

[0010] According to the first aspect of the embodiment of the present application, a container log asynchronous streaming method based on thread pool and queue is provided.

[0011] In one embodiment, the container log asynchronous streaming method based on thread pool and queue comprises:

[0012] The HTTP request parameter for obtaining the target container log initiated by the client is received and parsed, and the parsed HTTP request parameter is processed to generate a log production task, wherein the log production task comprises: target container name, session identifier, thread-safe queue and thread event signal;

[0013] Based on the pre-configured thread pool, the log production task is received, the log stream of the log production task is obtained through the container engine and preprocessed, and the preprocessed log stream is transmitted to the thread-safe queue for storage;

[0014] The thread-safe queue is transmitted to the pre-configured HTTP response header in block encoding format, flushed to the client, and it is judged whether the HTTP response header detects a termination signal or a thread event signal;

[0015] The connection state of the client is monitored, and if the connection is disconnected, the thread event signal is set and the log stream is closed and the thread-safe queue is emptied.

[0016] In one embodiment, the HTTP request parameter for obtaining the target container log initiated by the client is received and parsed, and the parsed HTTP request parameter is processed to generate a log production task, wherein the log production task comprises: target container name, session identifier, thread-safe queue and thread event signal comprises:

[0017] The HTTP request parameter for obtaining the target container log initiated by the client is received, and the request parameter is converted into a standardized format by using a request processor;

[0018] verify the validity of the standardized request parameters, and generate a target container name and a session identifier;

[0019] Initialize the standardized request parameters using the concurrent components built into the data standard library, generate a thread-safe queue and a thread event signal.

[0020] In one embodiment, the verification of the validity of the standardized request parameters, and the generation of the target container name and the session identifier include:

[0021] Check whether the standardized request parameters are within a preset threshold;

[0022] According to the business logic, the standardized request parameters are spliced according to specific rules and separators to generate a target container name;

[0023] Use the character identification library built into the data standard library to generate a session identifier for subsequent resource management and tracking.

[0024] In one embodiment, the receiving of the log production task based on the pre-configured thread pool, the obtaining of the log stream of the log production task through the container engine and the preprocessing, and the transmission of the preprocessed log stream to the thread-safe queue for storage include:

[0025] Use the container controller to transmit the received log production task to the pre-configured thread pool through the entry function for execution;

[0026] Obtain the log stream of the log production task through the container engine, and read the log data block from the log stream in a streaming manner;

[0027] Decode and split the log data block, and transmit the processed log data block to the thread-safe queue for storage. If the thread-safe queue is full, store it in the remote database.

[0028] In one embodiment, the decoding and splitting of the log data block, and the transmission of the processed log data block to the thread-safe queue for storage, if the thread-safe queue is full, the storage in the remote database include:

[0029] Decode the log data block using character encoding to prevent abnormal byte stream in the container log from causing the entire session to crash;

[0030] Split the log data block using character line splitting to ensure that any log data block can be correctly split;

[0031] Transmit the decoded and split log data block to the thread-safe queue for storage. When the thread-safe queue is full, temporarily store the decoded and split log data block in the remote database.

[0032] In one embodiment, the transmitting the thread-safe queue in chunked encoding format to the pre-configured HTTP response header, flushing to the client, and determining whether the HTTP response header detects a termination signal or a thread event signal comprises:

[0033] In one transmission cycle, the log content is obtained from the thread-safe queue by the asynchronous function in chunked transfer encoding format through the pre-configured HTTP response header;

[0034] The client transmits the obtained log content to the pre-configured HTTP response header in chunked transfer encoding format, and flushes to the client to realize real-time push;

[0035] When the HTTP response header detects a termination signal or a thread event signal, the cycle is terminated; if no termination signal and thread event signal are detected, the cycle is repeatedly continued to continuously transmit the log content.

[0036] In one embodiment, the monitoring the connection state of the client, if the connection is disconnected, setting a thread event signal and closing the log stream and emptying the thread-safe queue comprises:

[0037] The connection disconnection event of the client is listened to, if the connection is disconnected, a thread event signal is set to notify the thread-safe queue to stop working;

[0038] The corresponding log stream is actively searched and closed to stop the production of the log stream from the root;

[0039] The thread-safe queue is emptied of all residual log data blocks that have not been consumed, and part of the memory resources is released.

[0040] In one embodiment, the listening to the connection disconnection event of the client, if the connection is disconnected, setting a thread event signal to notify the thread-safe queue to stop working comprises:

[0041] After the client actively disconnects, the transmission control protocol connection is automatically detected to be closed by using the detection framework built in the data standard library;

[0042] A callback function is triggered for callback, a thread event signal is set, and the thread-safe queue is notified to stop working.

[0043] In one embodiment, the actively searching for and closing the corresponding log stream to stop the production of the log stream from the root comprises:

[0044] According to the target container name and the session identifier, the corresponding log stream object is searched from the log stream;

[0045] Forcibly close the log stream object and terminate the underlying container engine call to stop the production of log data at the source.

[0046] According to a second aspect of the embodiments of the present application, a container log asynchronous streaming system based on a thread pool and a queue is provided.

[0047] In one embodiment, a container log asynchronous streaming system based on a thread pool and a queue comprises:

[0048] A request processing module is configured to receive and parse a hypertext transfer protocol request parameter initiated by a client to obtain a target container log, and process the parsed hypertext transfer protocol request parameter to generate a log production task, wherein the log production task comprises a target container name, a session identifier, a thread-safe queue, and a thread event signal.

[0049] A log production module is configured to receive the log production task based on a pre-configured thread pool, obtain a log stream of the log production task through a container engine and perform preprocessing, and transmit the preprocessed log stream to a thread-safe queue for storage.

[0050] A streaming response module is configured to transmit the thread-safe queue to a pre-configured hypertext transfer protocol response header in a chunked encoding format, flush to a client, and determine whether the hypertext transfer protocol response header detects a termination signal or a thread event signal.

[0051] A connection management module is configured to monitor a connection state of the client, and if the connection is disconnected, set the thread event signal and close the log stream and empty the thread-safe queue.

[0052] According to a third aspect of the embodiments of the present application, a computer device is provided.

[0053] In some embodiments, the computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0054] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided.

[0055] In one embodiment, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0056] The technical solutions provided by the embodiments of the present application can include the following beneficial effects:

[0057] 1) The application adopts the mode of separating producers and consumers, and the log can be consumed and sent as soon as it is produced, without waiting for the completion of the acquisition of all logs, which greatly reduces the delay perceived by the client and realizes real-time log streaming.

[0058] 2) The application uses a queue with limited capacity as a buffer, effectively controlling the memory occupation of a single request. The thread pool avoids the risk of unlimited thread creation for each request, making the system resource utilization higher and more controllable.

[0059] 3) The asynchronous and non-blocking design of the application makes the main server process easily handle a large number of concurrent log requests, significantly improves the system throughput, and has good scalability.

[0060] 4) The application can timely and completely release all resources (threads, stream objects, queue memory) that are no longer needed due to client disconnection through the stop_event signal and active resource cleaning mechanism, avoiding resource leakage and improving system stability.

[0061] 5) The application provides a degradation strategy (such as storing in Redis) when the queue is full, ensuring that log data will not be completely lost in extreme cases, enhancing the reliability of the system.

[0062] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0063] The drawings incorporated into the specification and forming part of the specification, show embodiments consistent with the application, and together with the specification, serve to explain the principles of the application.

[0064] Figure 1 is one of the flowcharts of a container log asynchronous streaming method based on thread pool and queue according to an exemplary embodiment;

[0065] Figure 2 is a principle block diagram of a container log asynchronous streaming system based on thread pool and queue according to an exemplary embodiment;

[0066] Figure 3 is a structural schematic diagram of a computer device according to an exemplary embodiment;

[0067] Figure 4 is an architecture diagram of a container log asynchronous streaming system based on thread pool and queue according to an exemplary embodiment;

[0068] Figure 5Figure 2 is a flow chart of a container log asynchronous streaming method based on a thread pool and a queue according to an exemplary embodiment;

[0069] Figure 6 Figure 3 is a flow chart of connection interruption processing in a container log asynchronous streaming method based on a thread pool and a queue according to an exemplary embodiment. DETAILED DESCRIPTION

[0070] The following description and drawings are illustrative of specific embodiments thereof and are not intended to limit the scope of the embodiments. Parts and features of some embodiments can be included in, or alternative parts and features of other embodiments can be substituted for, parts and features of the other embodiments. The range of the embodiments herein includes the full range of equivalents to the claims and all available written descriptions of the embodiments. Various embodiments are described in a progressive manner, each focusing on the differences from other embodiments, and the same or similar parts between various embodiments can be mutually referred to.

[0071] Each module in the device or system of the present application can be realized wholly or partially by software, hardware, and a combination thereof. The above modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform the operations corresponding to the above modules.

[0072] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0073] Figure 1 An embodiment of a container log asynchronous streaming method based on a thread pool and a queue is shown.

[0074] In this optional embodiment, the container log asynchronous streaming method based on a thread pool and a queue comprises:

[0075] Step S101, receiving and parsing a hypertext transfer protocol (HTTP) request parameter initiated by a client to obtain a target container log, and processing the parsed HTTP request parameter to generate a log production task, wherein the log production task comprises a target container name, a session identifier (ID), a thread-safe queue (log_queue), and a thread event signal (stop_event);

[0076] Step S102, receiving the log production task based on a pre-configured thread pool, obtaining a log stream (log_stream) of the log production task through a container engine (such as DockerSDK) and performing preprocessing, and transmitting the preprocessed log stream to the thread-safe queue for storage;

[0077] Step S103, the thread-safe queue is transmitted to the pre-configured hypertext transfer protocol response header in block encoding format, refreshed to the client, and it is judged whether the hypertext transfer protocol response header detects a termination signal (None) or a thread event signal;

[0078] Step S104, the connection state of the client is monitored, if the connection is disconnected, the thread event signal is set and the log stream is closed and the thread-safe queue is emptied.

[0079] Specifically, a container log acquisition request initiated by a client is received; a thread-safe message queue and a stop event signal are created; a log production task is submitted to a predefined thread pool, and a working thread in the pool acquires a log stream from a container engine in real time, decodes and processes the log stream, and stores the log data in the message queue; a block transmission encoding HTTP response header is set, and an asynchronous coroutine is started to continuously acquire data from the message queue, and the data is sent to the client in a streaming transmission mode; the connection state of the client is listened to, and when a connection interruption is detected, the stop event signal is immediately set to notify the production thread to stop working, and a resource cleaning operation of closing the log stream and emptying the message queue is performed.

[0080] In this optional embodiment, the hypertext transfer protocol request parameters initiated by the client to acquire the target container log are received and parsed, and the parsed hypertext transfer protocol request parameters are processed to generate a log production task, wherein the log production task includes: a target container name, a session identifier, a thread-safe queue and a thread event signal.

[0081] The hypertext transfer protocol request parameters initiated by the client to acquire the target container log are received, and the request parameters are converted into a standardized (json) format by using a request handler (GetContainerLogHandler);

[0082] The validity of the standardized request parameters is verified, and a target container name and a session identifier are generated;

[0083] The standardized request parameters are initialized by using a concurrent component (threading\queue) built in a data standard library (python standard library) to generate a thread-safe queue and a thread event signal.

[0084] Specifically, as Figure 5As shown, S01, the client sends an HTTP POST request to the server, requesting to get the logs of a specified container. S02, the GetContainerLogHandler receives the request, parses the request body (JSON format), verifies the parameter legality, and generates the name of the target container and a globally unique session ID according to the parameters. In the GetContainerLogHandler.post method, the request body is parsed into a Python dictionary (data) using json.loads(self.request.body). The required parameters are verified: check if the required keys (such as 'type', 'user') exist in the dictionary using if not "type" in data:. If not, immediately set the HTTP status code to 400 and return an error message. The parameters include at least the container identifier, and can also include the log line limit (tail), the start time (since), the end time (until), and the follow flag. Two core resources are created for this request session: a thread-safe queue (log_queue) and a thread event signal (stop_event). S03, the concurrent components provided by the python standard library (threading\queue) are used. Create a queue: create a first-in-first-out (FIFO) queue with a maximum capacity of 50 using queue.Queue(maxsize=50). The maxsize parameter is used to implement back pressure and limit memory usage. Create an event signal: create a thread event object using threading.Event(). The initial state of the object is "unset", which can be set by the set() method, and its state can be queried by the is_set() method.

[0085] In this optional embodiment, the verification of the validity of the standardized request parameters and the generation of the target container name and session identifier include:

[0086] Check if the standardized request parameters are within the preset threshold;

[0087] According to the business logic, the standardized request parameters are spliced according to a specific rule and a separator (such as -) to generate the target container name;

[0088] Use the character identification library (UUID library) built into the data standard library (python standard library) to generate a session identifier, which is used for subsequent resource management and tracking.

[0089] Specifically, verify parameter validity: check if the parameter value is within the allowed range. For example, check if data["type"] exists in the predefined constant list constants.AVAILABLE_TASK_TYPE. Generate container name: according to the business logic, splice multiple parameters (user, type, id, project, task_name) according to certain rules and separators (such as -), to generate a container name (self.container_name=f"{user}{mid_name}{type}...") that is unique in the system. Generate session ID: use the UUID library (such as Python's uuid.uuid4().hex) to generate a globally unique string as the ID of this log session (self.id=uuid.uuid4().hex), which is used for subsequent resource management and tracking.

[0090] In this optional embodiment, the thread pool based on pre-configuration receives log production tasks, obtains log stream of the log production tasks through the container engine and performs preprocessing, and transmits the preprocessed log stream to the thread-safe queue for storage, including:

[0091] The received log production tasks are transmitted to the pre-configured thread pool for execution by the container controller (RflysimContainerController) through the entry function (log_producer function);

[0092] The log stream of the log production tasks is obtained through the container engine, and log data blocks are read from the log stream in a streaming manner;

[0093] The log data blocks are decoded and processed by lines, and the processed log data blocks are transmitted to the thread-safe queue for storage, and if the thread-safe queue is full, the data blocks are stored in the remote database (Redis database).

[0094] Specifically, as Figure 5S04, the RflysimContainerController.log_producer method is submitted to the thread pool for execution along with relevant parameters (container name, queue, stop event, etc.). RflysimContainerController.log_producer is a core instance method in the container controller class (RflysimContainerController). Its name can be literally translated as "log producer method". Its core function is to interact with the container engine in a separate thread, continuously produce log data, and put it into the shared queue. It is the entry function of the producer thread. S05, the producer thread in the thread pool is activated, which obtains the log stream (log_stream) object of the specified container through the Docker SDK. This object provides a continuous stream of log data. S06, the producer thread enters a loop, continuously reads raw data blocks from the log stream, decodes and splits the lines, and then puts the processed log lines into the log_queue. Degradation strategy: if the queue is full at this time, the log is stored in the Redis database to prevent data loss and record a warning.

[0095] In this optional embodiment, the decoding and line splitting of the log data block, and the transmission of the processed log data block to the thread-safe queue for storage, if the thread-safe queue is full, storing in the remote database include:

[0096] Decoding the log data block using character encoding (UTF-8) to prevent abnormal byte streams in container logs from causing the entire session to crash;

[0097] Splitting the log data block using character line splitting (tail buffer + split) to ensure that any log data block can be correctly split;

[0098] Transmitting the decoded and line-split log data block to the thread-safe queue for storage, and when the thread-safe queue is full, temporarily storing the decoded and line-split log data block in the remote database.

[0099] Specifically, the log data block will be read from the log_stream in streaming mode. Decode and split the data block by line: use UTF-8, error handling strategy is errors='replace', prevent abnormal byte stream in container from causing the entire session to crash; use "tail buffer + split" mode to ensure that blocks of any length can be correctly split; after the loop ends, if there is still residual in the buffer, decode and enqueue again. Create a message queue for the split log text. If the queue is full, provide a degradation strategy (store the log in an external storage system such as Redis when the queue is full: when the queue throws a queue.Full exception, write the log to the redis list). When a stop signal (stop_event.set()) or the end of the log stream is detected, a termination signal (None) is placed in the queue, and then the log stream is closed and the resources are released.

[0100] In this optional embodiment, the thread-safe queue is transmitted to the pre-configured hypertext transfer protocol response header in block encoding format, flushed to the client, and it is determined whether the hypertext transfer protocol response header detects a termination signal or a thread event signal, comprising:

[0101] In one transmission cycle, the asynchronous function is used to obtain the log content from the thread-safe queue through the pre-configured hypertext transfer protocol response header in block transmission encoding format;

[0102] The client transmits the obtained log content to the pre-configured hypertext transfer protocol response header in block transmission encoding format, and flushes it to the client, realizing real-time push;

[0103] When the hypertext transfer protocol response header detects a termination signal or a thread event signal, the cycle is terminated; if no termination signal and thread event signal are detected, the cycle is repeated to continue transmitting the log content.

[0104] Specifically, as shown in Figure 5 S07, immediately set the HTTP response header, declare the transmission encoding as block transmission (Transfer-Encoding: chunked), and start the streaming response. S08, in one transmission cycle, asynchronously obtain the log content from the message queue. S09, write each block of log content obtained in block encoding format to the HTTP response body and flush it to the client. S10, the consumer coroutine determines whether a termination signal (None) is received or the stop_event is set (indicating that the connection has been interrupted) in each cycle. If yes, exit the loop; if no, continue the loop. S11, the producer thread places a termination signal (None) in the queue when a stop signal or the end of the log stream is detected, and then safely closes the log stream and exits the thread.

[0105] In this optional embodiment, the monitoring of the connection state of the client, if the connection is disconnected, setting the thread event signal and closing the log stream and emptying the thread-safe queue includes:

[0106] Listening to the connection close event (on_connection_close) of the client, if the connection is disconnected, setting the thread event signal to notify the thread-safe queue to stop working;

[0107] Proactively searching for and closing the corresponding log stream (log_stream), stopping the production of the log stream from the root;

[0108] Emptying all remaining log data blocks in the thread-safe queue that have not been consumed, releasing part of the memory resources.

[0109] Specifically, as shown in Figure 6 the connection close event (on_connection_close) of the client is listened to. If triggered: immediately set the stop signal (stop_event.set()), notify the producer thread to stop working. Proactively search for and close the corresponding container log stream (log_stream), stop the log production from the root. Global registration: after the producer thread successfully creates the log stream (log_stream), it will be stored in a global, thread-safe dictionary (self.log_streams). The key of this dictionary is a unique identifier, which is spliced by the container name and the session ID of this request (for example, f"{self.container_name}:{self.id}"). Search: when the connection close event is triggered, the processor generates this unique key using the same rule. Retrieve and remove: find the log_stream object corresponding to the key in the dictionary. If found, use the pop method to take it out and remove it from the dictionary. Close: call the close() method of the taken out log_stream object. This method will forcibly close the underlying connection with the DockerDaemon, thereby immediately interrupting the blocking forloginlog_stream: loop in the producer thread, and throwing an exception to make the thread exit. Clear the remaining unsent log data in the message queue, quickly release the memory resources.

[0110] When the producer thread detects the stop signal or the natural end of the log stream, it puts a termination signal (None) into the queue, and then safely closes the log stream and exits the thread.

[0111] In this optional embodiment, the monitoring of the connection close event of the client, if the connection is disconnected, setting the thread event signal to notify the thread-safe queue to stop working includes:

[0112] When the client actively disconnects, the Transmission Control Protocol (TCP) connection is automatically detected by the built-in detection framework (Tornado framework) of the data standard library (python standard library), and the callback function (on_connection_close) is triggered for callback, and a thread event signal is set to notify the thread safe queue to stop working.

[0113] The callback function (on_connection_close) is triggered for callback, and a thread event signal is set to notify the thread safe queue to stop working.

[0114] Specifically, the connection is disconnected: the client actively disconnects (for example, the user closes the browser tab). The Tornado framework automatically detects that the TCP connection is closed, and then triggers the on_connection_close callback method of the GetContainerLogHandler. The first step of the callback method is to immediately set the stop_event event. This signal will cross the thread to notify the producer thread to stop working.

[0115] In this optional embodiment, the active search and closing of the corresponding log stream, and the stopping of the production of the log stream from the root include:

[0116] According to the target container name and the session identifier, the corresponding log stream object is found from the log stream;

[0117] The log stream object is forcibly closed, and the underlying container engine call is terminated, so that the production of log data is stopped from the root.

[0118] Specifically, the resource is searched: according to the container name and the session ID of the current request, the corresponding log stream object is accurately found from the global dictionary log_streams. The log stream is closed: the log stream object found in the previous step is forcibly closed. This operation will immediately terminate the underlying Docker API call, and stop the production of log data from the root. The buffer is cleaned: all residual log data in the log_queue queue that has not been consumed is emptied, and the memory is immediately released. The cleaning is completed: all related resources are released, and the cleaning process is completed.

[0119] Figure 2 An embodiment of a container log asynchronous streaming system based on a thread pool and a queue is shown.

[0120] In this optional embodiment, the container log asynchronous streaming system based on a thread pool and a queue includes:

[0121] The request processing module 201 is configured to receive and parse the hypertext transfer protocol request parameter initiated by the client for obtaining the target container log, and process the parsed hypertext transfer protocol request parameter to generate a log production task, wherein the log production task comprises a target container name, a session identifier, a thread-safe queue and a thread event signal.

[0122] The log production module 202 is configured to receive the log production task based on a pre-configured thread pool, obtain a log stream of the log production task through a container engine, and pre-process the log stream, and transmit the pre-processed log stream to the thread-safe queue for storage.

[0123] The streaming response module 203 is configured to transmit the thread-safe queue to a pre-configured hypertext transfer protocol response header in a block encoding format, flush to the client, and judge whether the hypertext transfer protocol response header detects a termination signal or a thread event signal.

[0124] The connection management module 204 is configured to monitor the connection state of the client, and if the connection is disconnected, set the thread event signal and close the log stream and empty the thread-safe queue.

[0125] Specifically, the client request is received, parsed and session resources are initialized; thread pool instances are maintained and log production tasks are scheduled; running in the thread pool, configured to obtain and process log stream data from the container engine; the log is streamed to the client in a block transmission encoding manner; the client connection state is monitored and the resource cleaning mechanism is triggered when interrupted.

[0126] In order to facilitate the understanding of the above technical solutions of the present application, the above technical solutions of the present application are further described from the perspective of architecture and principle as follows:

[0127] As shown in Figure 4 , the following core components are mainly included: POST / v1 / scheduler / get_log: this is an HTTP request endpoint (API interface). The client uses the HTTP POST method to send a request to this address on the server to obtain the container log.

[0128] Tornado HTTP server: as an asynchronous web server, responsible for listening and receiving requests sent to the / v1 / scheduler / get_log interface.

[0129] GetContainerLogHandler: a dedicated request processor (Handler class). In addition to parsing the request, the key is to implement the on_connection_close method to capture the client connection disconnection event and trigger resource cleaning.

[0130] ThreadPoolExecutor: built with python standard library concurrent.futures.ThreadPoolExecutor, used to execute blocking I / O operations (such as calling Docker API), isolated from the main asynchronous event loop of Tornado.

[0131] log_queue: a thread-safe First-In-First-Out (FIFO) queue built with queue.Queue, used as a buffer channel between log producer threads and consumer coroutines.

[0132] RflysimContainerController: a high-level module encapsulating the logic of interacting with Docker Engine. It calls container APIs through Docker SDK, and its core method log_producer is responsible for producing log data.

[0133] log_streams: a dictionary structure maintained within the controller, used to store and manage all active container log stream (log_stream) objects, each bound to a unique session ID.

[0134] In an embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 3 The computer device includes a processor, a memory, and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store static information and dynamic information data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement the steps in the above method embodiments.

[0135] Those skilled in the art can understand that Figure 3 The structure shown in the above

[0136] In addition, the present application further provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0137] In addition, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above method embodiments.

[0138] A person of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the computer program can include the processes of the above embodiments. Any reference to memory, storage, database or other medium in each embodiment of the present application can include at least one of non-volatile and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0139] The present application is not limited to the structures described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A thread pool and queue based container log asynchronous streaming method, characterized in that, The application comprises the following steps: Receiving and analyzing the HTTP request parameters initiated by the client to obtain the log of the target container, and processing the analyzed HTTP request parameters to generate a log production task, wherein the log production task includes the target container name, session identifier, thread-safe queue and thread event signal; Based on the pre-configured thread pool, the log production task is received, the log stream of the log production task is obtained through the container engine and pre-processed, and the pre-processed log stream is transmitted to the thread-safe queue for storage; The thread-safe queue is transmitted to the pre-configured HTTP response header in block encoding format, flushed to the client, and it is judged whether the HTTP response header detects a termination signal or a thread event signal; Monitoring the connection state of the client, if the connection is disconnected, setting the thread event signal and closing the log stream and emptying the thread-safe queue; The thread-safe queue is transmitted to the pre-configured HTTP response header in block encoding format, flushed to the client, and it is judged whether the HTTP response header detects a termination signal or a thread event signal. In one transmission cycle, the asynchronous function is used to obtain the log content from the thread-safe queue through the pre-configured HTTP response header in block transmission encoding format; The client transmits the obtained log content to the pre-configured HTTP response header in block transmission encoding format, and flushes it to the client to realize real-time push; When the HTTP response header detects a termination signal or a thread event signal, the cycle is terminated; if no termination signal or thread event signal is detected, the cycle is repeated to continue transmitting the log content.

2. The container log asynchronous streaming method based on thread pool and queue according to claim 1, characterized in that, The application comprises the following steps: Receiving the HTTP request parameters initiated by the client to obtain the log of the target container, and converting the request parameters into a standardized format using a request processor; Verifying the validity of the standardized request parameters, and generating the target container name and session identifier; Using the built-in concurrent components of the data standard library to initialize the standardized request parameters to generate the thread-safe queue and thread event signal.

3. The thread pool and queue based container log asynchronous streaming method of claim 2, wherein, The application comprises the following steps: Checking whether the standardized request parameters are within the preset threshold; According to the business logic, the standardized request parameters are spliced according to the specific rules and separators to generate the target container name; Using the character identification library built in the data standard library to generate the session identifier for subsequent resource management and tracking.

4. The thread pool and queue based container log asynchronous streaming method of claim 1, wherein, The application comprises the following steps: The container controller transmits the received log production task to a pre-configured thread pool for execution through an entry function; The container engine obtains a log stream of the log production task and reads log data blocks from the log stream in a streaming manner; The log data blocks are decoded and processed by line, and the processed log data blocks are transmitted to a thread-safe queue for storage. If the thread-safe queue is full, the log data blocks are stored in a remote database.

5. The thread pool and queue based container log asynchronous streaming method of claim 4, wherein, The decoding and line processing of the log data blocks and the storage of the processed log data blocks in the thread-safe queue include: The log data blocks are decoded and processed by character encoding to prevent abnormal byte streams in the container log from causing the entire session to crash; The log data blocks are processed by line to ensure that any log data block is correctly cut by line; The decoded and line-processed log data blocks are transmitted to the thread-safe queue for storage, and when the thread-safe queue is full, the decoded and line-processed log data blocks are temporarily stored in the remote database.

6. The thread pool and queue based container log asynchronous streaming method of claim 1, wherein, The monitoring of the connection state of the client includes: If the connection is disconnected, the thread event signal is set and the log stream is closed and the thread-safe queue is emptied. If the connection is disconnected, the thread event signal is set to notify the thread-safe queue to stop working. The corresponding log stream is actively searched and closed to stop the production of the log stream from the root.

7. The thread pool and queue based container log asynchronous streaming method of claim 6, wherein, The thread-safe queue is emptied of all remaining log data blocks that have not been consumed, releasing part of the memory resources. If the connection is disconnected, the thread event signal is set to notify the thread-safe queue to stop working. When the client actively disconnects, the Transmission Control Protocol connection is automatically detected by the built-in detection framework of the data standard library.

8. The thread pool and queue based container log asynchronous streaming method of claim 6, wherein, The callback function is triggered for callback, and the thread event signal is set to notify the thread-safe queue to stop working. The corresponding log stream is actively searched and closed to stop the production of the log stream from the root. The corresponding log stream object is found from the log stream according to the target container name and the session identifier.

9. A thread pool and queue based container log asynchronous streaming system, characterized in that, The log stream object is forcibly closed, and the underlying container engine call is terminated to stop the production of log data from the root. The request processing module receives and parses the Hyper Text Transfer Protocol request parameters initiated by the client to obtain the log of the target container, and processes the parsed Hyper Text Transfer Protocol request parameters to generate a log production task, wherein the log production task includes: target container name, session identifier, thread-safe queue, and thread event signal. The log production module receives the log production task based on the pre-configured thread pool, obtains the log stream of the log production task through the container engine, and pre-processes the log stream. The pre-processed log stream is transmitted to the thread-safe queue for storage. The streaming response module transmits the thread-safe queue to the pre-configured Hyper Text Transfer Protocol response header in a block encoding format, flushes it to the client, and judges whether the Hyper Text Transfer Protocol response header detects a termination signal or a thread event signal. The connection management module is used for monitoring the connection state of the client, setting a thread event signal and closing a log stream and emptying a thread-safe queue if the connection is disconnected; The transmission of the thread-safe queue to the preconfigured HTTP response header in the chunked encoding format, the flushing to the client, and the judgment of whether the HTTP response header detects the termination signal or the thread event signal include: In one transmission cycle, the asynchronous function is used to obtain the log content from the thread-safe queue in the chunked transmission encoding format through the preconfigured HTTP response header; The client transmits the obtained log content to the preconfigured HTTP response header in the chunked transmission encoding format, and flushes to the client to realize real-time pushing. When the HTTP response header detects the termination signal or the thread event signal, the cycle is terminated; if the termination signal and the thread event signal are not detected, the cycle is repeatedly continued to continuously transmit the log content.

Images