Large-scale high-throughput task query request processing method and system

Abstract

The invention discloses a large-scale high-throughput task query request processing method and system, and relates to the field of computer cluster scheduling. The large-scale high-throughput task query request processing system comprises a system architecture design module, a core processing mechanism module, a key configuration parameter setting module and a service operation process module. The large-scale high-throughput task query request processing system has the following technical advantages that the scheduling performance is improved through an architecture decoupling and resource isolation mechanism; the performance interference of lock competition, thread switching and frequent I / O operation in a traditional scheme on a scheduler is eliminated through architecture decoupling, the actual measurement scheduling delay is reduced by 40%-60%, the process creation overhead is reduced through a main control process merging request mechanism and maxrequest sonfork parameter regulation and control, the system throughput is improved, the resource utilization rate is optimized, and the system performance is improved. The dual memory cleaning strategy periodically cleans and keeps the timeliness to balance the query real-time performance and memory resource occupation, and the completed task is prevented from residing for a long time.

Description

Technical Field

[0001] This invention relates to the field of waste mineral oil, specifically a method and system for processing large-scale, high-throughput task query requests. Background Technology

[0002] The core computational tasks of Electronic Design Automation (EDA), including chip design verification, pre-tape-out simulation, and industry software testing, must be executed through distributed computing clusters. Typical characteristics include: Task decomposition: EDA workflows break down large-scale tasks into thousands to tens of thousands of short-duration subtasks for concurrent execution. High-frequency interaction: Multiple EDA workflows query task status at a rate of seconds to trigger subsequent task delivery. Scale requirements: Daily task submissions reach millions, peak query requests reach hundreds of times per second, and a single query request involves thousands of tasks.

[0003] Current mainstream cluster scheduling systems adopt a task scheduling and task query separation architecture, but they face the following problems in EDA scenarios: Lock contention bottleneck: In the existing architecture, the query service and the scheduler share the database. High-concurrency queries lead to database lock contention, which in turn increases scheduling latency; Resource contention defect: When using a multi-process / thread isolation scheme, the main control process frequently performs fork operations or thread switching, causing the core scheduling logic to be blocked; Data consistency risk: The state synchronization mechanism between the query service and the scheduler is insufficient, which may lead to misjudgment of task status.

[0004] Therefore, those skilled in the art have provided a method and system for processing large-scale, high-throughput task query requests to solve the problems mentioned in the background art. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for processing large-scale, high-throughput task query requests, in order to address the following problems encountered by current mainstream cluster scheduling systems in EDA scenarios, which adopt a task scheduling and task query separation architecture: Lock contention bottleneck: In the existing architecture, the query service and the scheduler share the database, and high-concurrency queries lead to database lock contention, thereby increasing scheduling latency; Resource contention defect: When using a multi-process / thread isolation scheme, the main control process frequently performs fork operations or thread switching, causing the core scheduling logic to be blocked; Data consistency risk: The state synchronization mechanism between the query service and the scheduler is insufficient, which may lead to misjudgment of task status.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A method and system for processing large-scale, high-throughput task query requests are disclosed. The method and system include a system architecture design module, a core processing mechanism module, a key configuration parameter setting module, and a service operation flow module. The system architecture design module constructs an independent query service, decoupling the task status query function from the task scheduler to form the following collaborative mechanism: a task scheduler, an independent query service, and a transparent query interface. The task scheduler is dedicated to task scheduling and resource allocation, maintaining two types of persistent data to ensure system fault recovery capabilities. The independent query service is decoupled from the task scheduler and is dedicated to handling high-frequency status query requests. The transparent query interface uses a unified command (bjobs) to achieve automatic service routing, automatically selecting the independent query service and task scheduler based on query parameters, maintaining EDA workflow interface compatibility. The core processing mechanism module includes a state update mechanism, high-concurrency processing optimization, and resource management mechanism. Through a request merging strategy, the main control process dynamically allocates child processes to handle requests based on the concurrency level. By default, a single child process can handle a maximum of 10 requests. Then, through an asynchronous processing mode, the child processes handle query requests in an asynchronous, non-blocking manner. After processing, they are automatically destroyed to release resources. Furthermore, through periodic cleanup, completed tasks are removed from memory by default every 10 minutes. Time-sensitive cleanup can also be selected, and completed tasks can be retained in memory for a maximum of 24 hours. Independent timers can also be used to trigger the cleanup. The memory data maintenance cycle is decoupled from the processing flow to avoid resource contention. Key configuration parameter setting module: Set the jobinfo_interval, jobclean_interval, keep_finishjob_period and max_requests_on_fork parameters, and set different function definitions, default values ​​and constraints for each. Service operation flow module: The default scheduling cycle of the independent query service is 5 seconds. Within each cycle, the entire chain of processing actions is completed, including task status updates, processing query requests, synchronizing tasks with the scheduler, and cleaning up memory of completed tasks. The task scheduler writes task status update events to the shared event stream cb.stream module in real time. The independent query service module reads incremental events and updates the local task status. The main control process continuously listens for client TCP connection requests and receives established connections in batches. It dynamically creates sub-processing processes, which process client requests in an asynchronous non-blocking mode. The maximum concurrent processing capacity of a single process is M query requests, which is configurable and defaults to 10. After completing request processing, the process is actively destroyed to release resources. The memory cleanup task is triggered by an independent timer, which defaults to 10 minutes. It evicts completed task objects that have resided in memory for more than 24 hours. The scheduler task information is synchronized every hour, and a full data consistency check is performed.

[0007] As a preferred embodiment of the present invention: in the system architecture design module, the task scheduler includes a cb.data module and a cb.stream module. The cb.data module is used to record task metadata for historical query and statistical analysis. The cb.stream module records task lifecycle events in append mode to realize the three-part process of submission, execution and completion.

[0008] In a preferred embodiment of the present invention: the system architecture design module includes an independent query service comprising an event-driven update module, a full synchronization module, and a dual data source loading mechanism module. The event-driven update module monitors changes to the cb.stream module file to obtain task status in real time, avoiding direct access to the task scheduler module. The full synchronization module synchronizes the memory task status from the task scheduler module at a preset period to ensure eventual data consistency, with the preset period defaulting to 1 hour. The dual data source loading mechanism module loads historical data from the cb.data module upon startup and obtains incremental updates through the cb.stream module during runtime.

[0009] As a preferred embodiment of the present invention: In the core processing mechanism module, when performing the status update mechanism, the independent query service captures incremental events of the cb.stream module in real time through file listening, updates the local task status, and the full synchronization module obtains the memory task status from the task scheduler module every hour and performs data consistency verification.

[0010] As a preferred embodiment of the present invention: In the key configuration parameter setting module, the jobinfo_interval parameter is defined as the time interval for synchronizing the task status with the task scheduler. This mechanism ensures the consistency of task status between the independent query service and the task scheduler. The default value of the jobinfo_interval parameter is 3600 seconds per hour, and the minimum is 600 seconds per 10 minutes. If the setting is too short, it may cause excessive load on the task scheduler.

[0011] As a preferred embodiment of the present invention: In the key configuration parameter setting module, the jobclean_interval parameter is defined as the memory task cleanup cycle. This mechanism can prevent memory resources from being wasted due to the long-term retention of completed tasks and improve system efficiency. The default value of the jobclean_interval parameter is 600 seconds every 10 minutes.

[0012] As a preferred embodiment of the present invention: In the key configuration parameter setting module, the function of the keep_finishjob_period parameter is defined as the retention time of completed tasks in memory. This parameter can balance the needs of task status query and memory resource usage, and avoid memory overflow. The default value of the keep_finishjob_period parameter is 86400 seconds every 24 hours.

[0013] As a preferred embodiment of the present invention: the max_requests_on_fork parameter is defined as the upper limit of requests that a single child process can handle. This mechanism can reduce the overhead of the main control process forking child processes and avoid blocking the core logic. The default value of the max_requests_on_fork parameter is 10.

[0014] Compared with the prior art, the beneficial effects of the present invention are: This invention provides a method and system for processing large-scale, high-throughput task query requests. Through architectural decoupling and resource isolation mechanisms, it achieves the following technical advantages: Improved scheduling performance: Architectural decoupling eliminates the performance interference of lock contention, thread switching, and frequent I / O operations on the scheduler in traditional solutions, resulting in a 40%-60% reduction in actual scheduling latency. A master process request merging mechanism: The `max_requests_on_fork` parameter reduces process creation overhead, improves system throughput, optimizes resource utilization, and employs a dual memory cleanup strategy—periodic cleanup and time-sensitive retention—to balance query real-time performance with memory resource consumption, preventing completed tasks from remaining resident for extended periods. Data consistency assurance: Incremental updates to the event stream combined with full synchronization ensure eventual consistency of task states. Compatibility and scalability: Transparent query commands (bjobs) maintain EDA workflow interface compatibility, and parameterized configuration supports adaptive expansion from small-to-medium-sized clusters to high-performance computing environments. Attached Figure Description

[0015] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a block diagram showing the components of an independent query service and a task scheduler in a method and system for processing query requests for large-scale, high-throughput tasks.

[0016] Figure 2 This is a schematic diagram illustrating the service operation process of a large-scale, high-throughput task query request processing method and an independent query service within the system. Detailed Implementation

[0017] In this embodiment of the invention, a method and system for processing large-scale, high-throughput task query requests are disclosed. The method and system include a system architecture design module, a core processing mechanism module, a key configuration parameter setting module, and a service operation flow module. The system architecture design module constructs an independent query service, decoupling the task status query function from the task scheduler to form the following collaborative mechanism: a task scheduler, an independent query service, and a transparent query interface. The task scheduler is dedicated to task scheduling and resource allocation, maintaining two types of persistent data to ensure system fault recovery capabilities. The independent query service is decoupled from the task scheduler and is dedicated to processing high-frequency status query requests. The transparent query interface uses the unified command `bjobs` to achieve automatic service routing, automatically selecting the independent query service and task scheduler based on query parameters, maintaining EDA workflow interface compatibility. The core processing mechanism module includes a state update mechanism, high-concurrency processing optimization, and resource management mechanism. Through a request merging strategy, the main control process dynamically allocates child processes to handle requests based on the concurrency level. By default, a single child process can handle a maximum of 10 requests. Then, through an asynchronous processing mode, the child processes handle query requests in an asynchronous, non-blocking manner. After processing, they are automatically destroyed to release resources. Furthermore, through periodic cleanup, completed tasks are removed from memory by default every 10 minutes. Time-sensitive cleanup can also be selected, and completed tasks can be retained in memory for a maximum of 24 hours. Independent timers can also be used to trigger the cleanup. The memory data maintenance cycle is decoupled from the processing flow to avoid resource contention. Key configuration parameter setting module: Set the jobinfo_interval, jobclean_interval, keep_finishjob_period and max_requests_on_fork parameters, and set different function definitions, default values ​​and constraints for each. Service operation flow module: The default scheduling cycle of the independent query service is 5 seconds. Within each cycle, the entire chain of processing actions is completed, including task status updates, processing query requests, synchronizing tasks with the scheduler, and cleaning up memory of completed tasks. The task scheduler writes task status update events to the shared event stream cb.stream module in real time. The independent query service module reads incremental events and updates the local task status. The main control process continuously listens for client TCP connection requests and receives established connections in batches. It dynamically creates sub-processing processes, which process client requests in an asynchronous non-blocking mode. The maximum concurrent processing capacity of a single process is M query requests, which is configurable and defaults to 10. After completing request processing, the process is actively destroyed to release resources. The memory cleanup task is triggered by an independent timer, which defaults to 10 minutes. It evicts completed task objects that have resided in memory for more than 24 hours. The scheduler task information is synchronized every hour, and a full data consistency check is performed.

[0018] The following technical advantages are achieved through architectural decoupling and resource isolation mechanisms: Improved scheduling performance: Architectural decoupling eliminates the performance interference of lock contention, thread switching, and frequent I / O operations on the scheduler, resulting in a 40%-60% reduction in actual scheduling latency. A request merging mechanism for the main control process: The `max_requests_on_fork` parameter reduces process creation overhead, improves system throughput, and optimizes resource utilization. A dual memory cleanup strategy balances query real-time performance with memory resource consumption by periodically cleaning up and retaining time-sensitive data, preventing completed tasks from remaining resident for extended periods.

[0019] In the system architecture design module, the task scheduler includes the cb.data module and the cb.stream module. The cb.data module records task metadata for historical queries and statistical analysis, while the cb.stream module records task lifecycle events in append mode, implementing the three-part process of submission, execution, and completion. The independent query service in the system architecture design module includes an event-driven update module, a full synchronization module, and a dual-data source loading mechanism module. The event-driven update module listens for file changes in the cb.stream module to obtain task status in real time, avoiding direct access to the task scheduler module. The full synchronization module synchronizes the in-memory task status from the task scheduler module at a preset period, ensuring eventual data consistency; the preset period defaults to 1 hour. The dual-data source loading mechanism module loads historical data from the cb.data module at startup and obtains incremental updates from the cb.stream module during runtime. In the core processing mechanism module, during the status update mechanism, the independent query service captures incremental events from the cb.stream module in real time through file listening to update the local task status. The full synchronization module obtains the in-memory task status from the task scheduler module every hour and performs data consistency checks. In the key configuration parameter settings module, the `jobinfo_interval` parameter defines the time interval for synchronizing task status with the task scheduler. This mechanism ensures the consistency of task status between the independent query service and the task scheduler. The default value of `jobinfo_interval` is 3600 seconds per hour, and the minimum is 600 seconds per 10 minutes. Setting it too short may cause excessive load on the task scheduler. In the same module, the `jobclean_interval` parameter defines the memory task cleanup cycle. This mechanism prevents memory resources from being wasted by retaining completed tasks for extended periods, improving system efficiency. The default value of `jobclean_interval` is 600 seconds per 10 minutes. Finally, the `keep_finishjob_period` parameter defines the retention time of completed tasks in memory. This parameter balances task status query needs with memory resource usage, preventing memory overflow. The default value of `keep_finishjob_period` is 86400 seconds per 24 hours. The `max_requests_on_fork` parameter defines the maximum number of requests a single child process can handle. This mechanism reduces the overhead of the main process forking child processes and avoids blocking core logic. The default value of the `max_requests_on_fork` parameter is 10.

[0020] Data consistency is guaranteed through a combination of incremental updates and full synchronization of event streams, ensuring eventual consistency of task states. Compatibility and scalability are ensured through transparent query commands (bjobs) that maintain EDA workflow interface compatibility, while parameterized configuration supports adaptable scaling from small- to medium-sized clusters to high-performance computing environments.

[0021] The working principle of this invention is as follows: A method and system for processing large-scale, high-throughput task query requests includes a system architecture design module, a core processing mechanism module, a key configuration parameter setting module, and a service operation process module. The system architecture design module constructs an independent query service, decoupling the task status query function from the task scheduler to form the following collaborative mechanism: a task scheduler, an independent query service, and a transparent query interface. The task scheduler is dedicated to task scheduling and resource allocation, maintaining two types of persistent data to ensure system fault recovery capabilities. The independent query service is decoupled from the task scheduler and is dedicated to handling high-frequency status query requests. The transparent query interface uses the unified command `bjobs` to achieve automatic service routing, automatically selecting the independent query service and task scheduler based on query parameters, maintaining EDA workflow interface compatibility. In the system architecture design module, the task scheduler includes a `cb.data` module and a `cb.stream` module. The `cb.data` module records task metadata for historical querying and statistical analysis, while the `cb.stream` module records task lifecycle events in append mode, implementing the three-part process of submission, execution, and completion. In the system architecture design module, the independent query service includes an event-driven update module, a full synchronization module, and a dual data source loading mechanism module. The event-driven update module listens for changes in the cb.stream module file to obtain the task status in real time, avoiding direct access to the task scheduler module. The full synchronization module synchronizes the memory task status from the task scheduler module at a preset period to ensure eventual data consistency; the preset period is 1 hour by default. The dual data source loading mechanism module loads historical data from the cb.data module at startup and obtains incremental updates through the cb.stream module at runtime.

[0022] The core processing mechanism module includes a state update mechanism, high-concurrency processing optimization, and resource management mechanism. Through a request merging strategy, the main control process dynamically allocates child processes to handle requests based on concurrency levels. The default maximum number of requests handled by a single child process is 10. Then, through asynchronous processing, child processes handle query requests in an asynchronous, non-blocking manner. After processing, they are automatically destroyed to release resources. Periodic cleanup is also implemented, removing completed tasks from memory by default every 10 minutes. Time-sensitive cleanup is also available, with a maximum retention time of 24 hours for completed tasks in memory. Independent timers can also be used to trigger this process. The memory data maintenance cycle is decoupled from the processing flow, avoiding resource contention. Within the core processing mechanism module, during the state update mechanism, the independent query service captures incremental events from the cb.stream module in real time through file listening, updating the local task status. The full synchronization module obtains the memory task status from the task scheduler module every hour and performs data consistency checks.

[0023] The key configuration parameter setting module allows setting the `jobinfo_interval`, `jobclean_interval`, `keep_finishjob_period`, and `max_requests_on_fork` parameters, each with different functional definitions, default values, and constraints. Within this module, `jobinfo_interval` defines the time interval for synchronizing task status with the task scheduler. This mechanism ensures consistency of task status between the independent query service and the task scheduler. The default value for `jobinfo_interval` is 3600 seconds per hour, with a minimum of 600 seconds per 10 minutes. Setting it too short may cause excessive load on the task scheduler. Similarly, `jobclean_interval` defines the memory task cleanup cycle. This mechanism prevents memory resources from being wasted by retaining completed tasks for extended periods, improving system efficiency. The default value for `jobclean_interval` is 600 seconds per 10 minutes. In the key configuration parameter settings module, the `keep_finishjob_period` parameter defines the duration for which completed tasks are retained in memory. This parameter balances the need for task status queries with memory resource usage, preventing memory overflow. The default value for `keep_finishjob_period` is 86400 seconds every 24 hours. The `max_requests_on_fork` parameter defines the maximum number of requests a single child process can handle. This mechanism reduces the overhead of the main control process forking child processes, preventing blocking of core logic. The default value for `max_requests_on_fork` is 10.

[0024] Service operation flow module: The default scheduling cycle of the independent query service is 5 seconds. Within each cycle, the entire chain of processing actions is completed, including task status updates, processing query requests, synchronizing tasks with the scheduler, and cleaning up memory of completed tasks. The task scheduler writes task status update events to the shared event stream cb.stream module in real time. The independent query service module reads incremental events and updates the local task status. The main control process continuously listens for client TCP connection requests and receives established connections in batches. It dynamically creates sub-processing processes, which process client requests in an asynchronous non-blocking mode. The maximum concurrent processing capacity of a single process is M query requests, which is configurable and defaults to 10. After completing request processing, the process is actively destroyed to release resources. The memory cleanup task is triggered by an independent timer, which defaults to 10 minutes. It evicts completed task objects that have resided in memory for more than 24 hours. The scheduler task information is synchronized every hour, and a full data consistency check is performed.

[0025] The following technical advantages are achieved through architectural decoupling and resource isolation mechanisms: Improved scheduling performance: Architectural decoupling eliminates the performance interference of lock contention, thread switching, and frequent I / O operations on the scheduler, resulting in a 40%-60% reduction in actual scheduling latency. Master process request merging mechanism: The `max_requests_on_fork` parameter reduces process creation overhead, improves system throughput, and optimizes resource utilization. A dual memory cleanup strategy balances query real-time performance with memory resource consumption, preventing completed tasks from remaining resident for extended periods. Data consistency guarantee: Incremental updates and full synchronization of event streams ensure eventual consistency of task states. Compatibility and scalability: Transparent query commands (bjobs) maintain EDA workflow interface compatibility, and parameterized configuration supports adaptive expansion from small-to-medium-sized clusters to high-performance computing environments.

[0026] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method and system for processing large-scale high-throughput task query requests, characterized in that, The large-scale high-throughput task query request processing method and system comprises a system architecture design module, a core processing mechanism module, a key configuration parameter setting module and a service running flow module: The system architecture design module: constructs an independent query service, decouples the task state query function from the task scheduler, forms the following collaborative mechanism, which comprises a task scheduler, an independent query service and a transparent query interface, the task scheduler is responsible for task scheduling and resource allocation, maintains two types of persistent data and guarantees system fault recovery capability; the independent query service is decoupled from the task scheduler and is used for processing high-frequency state query requests; the transparent query interface realizes service automatic routing through a unified command bjobs, automatically selects the independent query service and the task scheduler according to the query parameters and maintains EDA workflow interface compatibility; The core processing mechanism module: the core processing mechanism module comprises a state updating mechanism, a high-concurrency processing optimization and a resource management mechanism, through a request merging strategy, the master process dynamically allocates sub-processes to process requests according to the concurrency, the default upper limit of a single sub-process is 10 requests, then through an asynchronous processing mode, the sub-processes process the query requests in an asynchronous non-blocking mode, after the processing is completed, the sub-processes are automatically destroyed to release resources, and then through periodic cleaning, the completed tasks in the memory are removed every 10 minutes, the time-sensitive cleaning can also be selected, the maximum retention time of the completed tasks in the memory is 24 hours, the independent timer trigger can also be performed, the memory data maintenance period is decoupled from the processing flow to avoid resource contention; The key configuration parameter setting module: the jobinfo_interval parameter, the jobclean_interval parameter, the keep_finishjob_period parameter and the max_requests_on_fork parameter are set, and different function definitions, default values and constraint conditions are set respectively; The service running flow module: the default scheduling period of the independent query service is 5 seconds, the full-link processing actions are completed in each period, including task state updating, processing query requests, synchronizing tasks with the scheduler and cleaning the memory of completed tasks, the task scheduler writes the task state updating events into a shared event stream cb.stream module in real time, the independent query service module reads the incremental events and updates the local task state, the master process continuously listens to the client TCP connection requests, receives the established connections in batches, dynamically creates sub-processing processes, the sub-processing processes process the client requests in an asynchronous non-blocking mode, the upper limit of single-process concurrency processing is M query requests, M is configurable and the default value is 10, after the request processing is completed, the process is actively destroyed to release resources, the memory cleaning task depends on the independent timer trigger, the default value is 10 minutes, the tasks that have been completed and have resided for more than 24 hours in the memory are eliminated, the task information of the scheduler is synchronized every hour, and the full-amount data consistency check is performed. 2.The method and system of claim 1, wherein, In the system architecture design module, the task scheduler includes a cb.data module and a cb.stream module, the cb.data module is used for recording task metadata, and is used for historical query and statistical analysis, and the cb.stream module records task life cycle events in an append mode, and realizes three part processes of submission, execution and completion. 3.The method and system of claim 1, wherein, In the system architecture design module, the independent query service includes an event-driven update module, a full-amount synchronization module and a double data source loading mechanism module, the event-driven update module listens to cb.stream module file changes to obtain task states in real time, and direct access to the task scheduler module is avoided; the full-amount synchronization module synchronizes memory task states from the task scheduler module according to a preset period, and ensures data final consistency, and the preset period is 1 hour by default; The double data source loading mechanism module loads historical data from the cb.data module when starting, and obtains incremental updates through the cb.stream module when running.

4. The method and system for processing large-scale high-throughput query requests according to claim 1, wherein, In the core processing mechanism module, when the state updating mechanism is performed, the independent query service captures incremental events of the cb.stream module in real time through file listening, and updates local task states, and the full-amount synchronization module obtains memory task states from the task scheduler module every hour, and performs data consistency verification.

5. The method and system for processing large-scale high-throughput query requests according to claim 1, wherein, In the key configuration parameter setting module, the function of the jobinfo_interval parameter is defined as a time interval for synchronizing task states with the task scheduler, this mechanism guarantees the task state consistency between the independent query service and the task scheduler, and the default value of the jobinfo_interval parameter is 3600 seconds per 1 hour, and the minimum is 600 seconds per 10 minutes, and if the setting is too short, the task scheduler may be subjected to high load.

6. The method and system for processing large-scale high-throughput query requests according to claim 1, wherein, In the key configuration parameter setting module, the function of the jobclean_interval parameter is defined as a memory task cleaning period, this mechanism can prevent memory resources from being wasted due to long-term retention of completed tasks, and system efficiency is improved, and the default value of the jobclean_interval parameter is 600 seconds per 10 minutes.

7. The method and system for processing large-scale high-throughput query requests according to claim 1, wherein, In the key configuration parameter setting module, the function of the keep_finishjob_period parameter is defined as a retention time length of completed tasks in memory, through this parameter, task state query demand and memory resource occupation can be balanced, memory overflow is avoided, and the default value of the keep_finishjob_period parameter is 86400 seconds per 24 hours.

8. The method and system for processing large-scale high-throughput query requests according to claim 1, wherein, The function of the max_requests_on_fork parameter is defined as a limit of single-subprocess request processing, this mechanism can reduce the overhead of the main control process forking a child process, and core logic blocking is avoided, and the default value of the max_requests_on_fork parameter is 10.

Images