Cooperative scheduling method for thread pool and cache in instantaneous high-concurrency scene

Abstract

The invention discloses a thread pool and cache co-scheduling method in an instantaneous high-concurrency scene, and relates to the field of data stream processing, and the method comprises the following steps: asynchronously receiving and decoding performance data through message-oriented middleware; performing standardization processing on the data object by using the parameterized thread pool; generating a dynamic routing weight based on multi-dimensional evaluation, and dividing the data objects into high, medium and low priorities according to the dynamic routing weight; writing high-priority data into a cache and storing the high-priority data into a batch queue according to the priority, only storing common-priority data into the queue, and discarding low-priority data; monitoring the batch queue, and triggering batch processing when a data volume threshold value or a time window threshold value is reached; asynchronously persisting the batch data to a column database; the query request is routed, real-time query is guided to the cache, and historical query is guided to the column database. And high-throughput, low-delay and high-reliability data processing under a high-concurrency scene is realized through cooperative scheduling and batch processing mechanisms of the thread pool and the cache.

Description

Technical Field

[0001] This invention relates to the field of data stream processing, specifically to a thread pool and cache collaborative scheduling method for instantaneous high-concurrency scenarios. Background Technology

[0002] With the rapid development of IoT and cloud computing technologies, intelligent operation and maintenance systems need to process high-frequency performance data from massive amounts of devices and subsystems. This type of data typically exhibits characteristics of large instantaneous traffic and data volumes reaching hundreds of millions, placing high demands on the real-time access, efficient storage, and fast query capabilities of data processing platforms. To address these challenges, various data processing solutions have emerged in existing technologies. Early solutions often directly used relational databases for real-time data writing and querying, offering strong transaction consistency, but prone to performance pressure under high-concurrency write scenarios. To further improve processing efficiency, the industry introduced message middleware to achieve asynchronous decoupling and employed thread pool technology to manage concurrent tasks, effectively alleviating the instantaneous pressure on the system. In recent years, to balance real-time performance with the need for large-scale data querying, a technical architecture combining memory caching and batch processing has been adopted. Real-time data access is provided through caching components such as Redis, and data is persisted to the database in batches, thereby reducing frequent database I / O operations. In addition, columnar databases such as ClickHouse, due to their excellent performance in batch writing and aggregate querying, are also gradually being applied to massive data storage and analysis scenarios. Summary of the Invention

[0003] Based on the shortcomings of the prior art described above, the purpose of this invention is to provide a thread pool and cache collaborative scheduling method for instantaneous high-concurrency scenarios, so as to solve the above-mentioned technical problems.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a thread pool and cache collaborative scheduling method for instantaneous high-concurrency scenarios, comprising: S1: Asynchronously receive performance data streams reported by external systems through message middleware, and deserialize and decode the received data messages to obtain structured data objects; S2: Use a parameterized thread pool to schedule and process structured data objects, and standardize the data objects to obtain standardized data objects; S3: Perform multi-dimensional evaluation on each standardized data object, generate dynamic routing weights, and assign corresponding priority tags to the data objects based on the comparison results of the dynamic routing weights and preset thresholds; S4: Perform differentiated data routing based on priority markers, write high-priority data objects to the cache and store them in the batch queue, store ordinary-priority data objects only in the batch queue, and discard low-priority data objects. S5: Monitor the status of the batch queue and trigger batch processing operations based on the dual conditions of queue data object backlog and maximum waiting time; S6: The batch data set retrieved from the batch queue after the trigger condition is met is asynchronously persisted to the columnar database in a single operation; S7: Identifies and routes received data query requests. Real-time query requests are directed to the cache to obtain the latest data, while query requests involving historical ranges or aggregation operations are directed to the columnar database for processing.

[0005] The present invention is further configured such that S1 includes: Configure the message middleware consumer client in the data processing service so that it can establish a connection with the specified message topic through a subscription mechanism; The consumer client continuously listens to the message topic through polling. When it detects that a new performance data message has been published, it retrieves the message content in an asynchronous and non-blocking manner. The obtained message body is parsed, and the message body is encoded using a predefined serialization format; Based on a predefined data schema specification, the serialized byte stream is converted into a structured data object that can be operated on internally by the service through deserialization operations.

[0006] The present invention is further configured such that S2 includes: Configure the thread pool executor, which includes a preset number of core threads, a maximum number of threads, and a bounded blocking queue with a fixed capacity; The decoded structured data object is encapsulated into an executable task, and the executable task is submitted to the thread pool executor through the submission method; After a task is submitted, if the core thread limit has not been reached, a new thread is created to process it. If the core thread limit has been reached, the task is stored in a bounded blocking queue for buffering. When the bounded blocking queue reaches its capacity limit, a new thread is started until the maximum number of threads is reached. When the number of threads reaches the maximum value and the queue is full, subsequent submitted tasks are processed according to the preset rejection policy. The tasks executed in the thread pool include the processing logic of data objects, including data validity verification, data format standardization processing, and supplementation of metadata information; The thread pool processes the data, transforming it from structured data objects to standardized data objects.

[0007] The present invention is further configured such that S3 includes: The routing decision-maker performs multi-dimensional evaluation of each standardized data object and generates dynamic routing weights for each data object. The multi-dimensional assessment includes data value assessment based on predefined parameter importance mapping, assessment by comparing the degree of change between current data and historical cached data, and system load assessment by real-time monitoring of system resource indicators; The dynamic routing weights are compared with the preset priority threshold range, and the data objects are divided into three levels: high priority, normal priority and low priority based on the comparison results. Based on the priority level classification, a corresponding priority tag is attached to each data object.

[0008] The present invention is further configured such that S4 includes: A differentiated data routing strategy is executed based on the priority tag attached to the data object, and the routing strategy is processed according to three priority levels respectively; For data objects marked as high priority, update operations in the cache database and add operations in the batch write queue are performed simultaneously. The cache update adopts a predetermined data structure with the device identifier as the primary key and the combination of unit identifier and parameter name as fields. For data objects marked as normal priority, a single routing operation is performed, directly adding the data object to the end of the batch write queue without performing an update operation on the cache database; For data objects marked as low priority, perform a data discard operation to terminate the subsequent processing flow of that data object.

[0009] The present invention is further configured such that S5 includes: The status changes of the batch queue are continuously monitored through a separately configured daemon thread or a timed task. Set dual trigger conditions for batch processing. The first trigger condition is that the backlog of ordinary priority data objects in the queue reaches a preset threshold. The second trigger condition is that the time interval since the last batch processing operation reaches a preset maximum time window. A first-come, first-served trigger mechanism is adopted, and batch processing operations are started immediately when any trigger condition is met first. During batch processing, data objects are extracted from the queue in descending order of their priority, with high-priority data objects being processed first and ordinary-priority data objects being processed subsequently. The extracted data objects are combined to form a batch data set.

[0010] The present invention is further configured such that S6 includes: Convert batch datasets into batch operation commands optimized for columnar databases; The entire batch of data can be submitted to a columnar database for storage through a single database interaction session. By leveraging the batch write capabilities of columnar databases, asynchronous non-blocking methods are used to complete data persistence operations.

[0011] The present invention is further configured such that S7 includes: Establish a unified query interface at the application service layer to receive external query requests; Perform feature analysis on the received query requests and determine the route based on whether the query conditions contain time range parameters or aggregation operation identifiers; For real-time query requests, they are routed to the cache database, and the latest values ​​of the specified device parameters are retrieved using key-value retrieval. For historical query requests, they are routed to a columnar database, where complex queries and data aggregation operations are performed using structured query statements.

[0012] The present invention is further configured such that the method also includes a fault tolerance mechanism, wherein when batch data persistence fails, the unsuccessful data objects are temporarily stored in a buffer and retried preferentially in the next batch processing cycle.

[0013] This invention provides a thread pool and cache collaborative scheduling method for instantaneous high-concurrency scenarios. The method comprises: S1: Asynchronously receiving performance data streams reported by external systems via a message middleware and deserializing and decoding the received data messages to obtain structured data objects; S2: Scheduling and processing the structured data objects using a parameterized thread pool and standardizing the data objects to obtain standardized data objects; S3: Performing multi-dimensional evaluation on each standardized data object to generate dynamic routing weights, and assigning corresponding priority tags to the data objects based on the comparison results of the dynamic routing weights and preset thresholds; S4: Performing differentiated data routing based on the priority tags. The process involves several steps: S1: Writing high-priority data objects to the cache and storing them in a batch queue; S2: Storing ordinary-priority data objects only in the batch queue; and S3: Discarding low-priority data objects. S4: Monitoring the status of the batch queue and triggering batch processing operations based on both the backlog of data objects and the maximum waiting time. S5: Asynchronously persisting the batch data set retrieved from the batch queue after the triggering conditions are met to the columnar database in a single operation. S6: Identifying and routing received data query requests. Real-time query requests are directed to the cache to obtain the latest data, while query requests involving historical ranges or aggregation operations are directed to the columnar database for processing. The resulting benefits include: 1. By using differentiated data routing and batch writing coordination mechanism, dynamic linkage between thread resources and cache resources is achieved. High-priority data is cached and batch synchronized in real time, ordinary data is processed with delay, and low-value data is intelligently discarded. This enables the layered elimination of processing pressure under instantaneous high concurrency, effectively alleviates the database write bottleneck, and improves system throughput and real-time response performance. 2. By using a dual triggering mechanism of backlog volume and time window, the timing of batch processing is adaptively adjusted, and data is automatically temporarily stored and prioritized for retry when write fails, forming a highly reliable persistence closed-loop mechanism, thereby maintaining business continuity and data consistency in the event of anomalies or database unavailability.

[0014] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 The flowchart illustrates a thread pool and cache collaborative scheduling method in a high-concurrency scenario, as an exemplary embodiment of the present invention. Detailed Implementation

[0016] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0017] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0018] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0019] A thread pool and cache coordinated scheduling method for instantaneous high concurrency scenarios, such as Figure 1 As shown, it includes: S1: Asynchronously receive performance data streams reported by external systems through message middleware, and deserialize and decode the received data messages to obtain structured data objects; S2: Use a parameterized thread pool to schedule and process structured data objects, and standardize the data objects to obtain standardized data objects; S3: Perform multi-dimensional evaluation on each standardized data object, generate dynamic routing weights, and assign corresponding priority tags to the data objects based on the comparison results of the dynamic routing weights and preset thresholds; S4: Perform differentiated data routing based on priority markers, write high-priority data objects to the cache and store them in the batch queue, store ordinary-priority data objects only in the batch queue, and discard low-priority data objects. S5: Monitor the status of the batch queue and trigger batch processing operations based on the dual conditions of queue data object backlog and maximum waiting time; S6: The batch data set retrieved from the batch queue after the trigger condition is met is asynchronously persisted to the columnar database in a single operation; S7: Identifies and routes received data query requests. Real-time query requests are directed to the cache to obtain the latest data, while query requests involving historical ranges or aggregation operations are directed to the columnar database for processing.

[0020] The present invention is further configured such that S1 includes: Configure the message middleware consumer client in the data processing service so that it can establish a connection with the specified message topic through a subscription mechanism; The consumer client continuously listens to the message topic through polling. When it detects that a new performance data message has been published, it retrieves the message content in an asynchronous and non-blocking manner. The obtained message body is parsed, and the message body is encoded using a predefined serialization format; According to predefined data pattern specifications, serialized byte streams are converted into structured data objects that can be operated internally by deserialization operations. Specifically, firstly, a persistent connection with the message middleware is established at the connection layer by configuring parameters such as server address and consumer group identifier, and the message subscription client is initialized. Then, at the message scheduling layer, a polling mechanism is used to continuously listen to a specified topic, and the receiving process is automatically triggered when a new performance data message is detected. The transport layer encodes the data using a predefined serialization format, and after the message format is verified by the built-in parser, the byte stream is converted into a structured data object containing business semantics according to the data structure mapping rules. During the conversion process, the system automatically supplements metadata such as data reception timestamps and source identifiers. The final standard data object fully includes core fields such as device identifier, unit number, parameter name and value, data generation and reception timestamps, and source address. The entire process runs in an asynchronous non-blocking mode, effectively isolating the production end and the consumer end by utilizing the decoupling characteristics of the message middleware, thereby achieving adaptive processing of instantaneous fluctuations in the data stream.

[0021] The present invention is further configured such that S2 includes: Configure the thread pool executor, which includes a preset number of core threads, a maximum number of threads, and a bounded blocking queue with a fixed capacity; The decoded structured data object is encapsulated into an executable task, and the executable task is submitted to the thread pool executor through the submission method; After a task is submitted, if the core thread limit has not been reached, a new thread is created to process it. If the core thread limit has been reached, the task is stored in a bounded blocking queue for buffering. When the bounded blocking queue reaches its capacity limit, a new thread is started until the maximum number of threads is reached. When the number of threads reaches the maximum value and the queue is full, subsequent submitted tasks are processed according to the preset rejection policy. The tasks executed in the thread pool include the processing logic of data objects, including data validity verification, data format standardization processing, and supplementation of metadata information; The thread pool processes the transformation from structured data objects to standardized data objects. Specifically, the data processing service is configured with a thread pool executor. This thread pool consists of a fixed number of core threads, a dynamically expandable maximum number of threads, and a blocking queue with a fixed capacity. The thread pool uses a fine-grained scheduling strategy to dynamically allocate and control traffic for high-concurrency tasks. During initialization, the number of core threads and the maximum number of threads are defined. Core threads are used for resident task processing to ensure basic throughput. The decoded structured data objects are encapsulated into independent tasks, each containing business logic such as data validity verification, data standardization processing, and metadata enhancement. Tasks are submitted to the thread pool executor through the submission interface provided by the thread pool, awaiting scheduling. The thread pool prioritizes using core threads to process tasks. When core threads are at full capacity, newly submitted tasks are placed in a fixed-capacity blocking queue for buffering. The upper limit of the blocking queue capacity is set to prevent unlimited memory growth and to provide temporary buffering for sudden traffic spikes. When the queue reaches its capacity limit... Within a time limit, the thread pool creates additional threads to process tasks as needed until the total number of threads reaches the maximum. If the maximum number of threads has been reached and the queue is full, subsequent tasks will be processed according to a preset rejection policy, such as being executed directly by the submitting thread or throwing an exception, thereby ensuring the stability and security of the system. In the thread pool, each worker thread obtains tasks in a first-in, first-out order and executes the corresponding data processing logic. During the execution of each task, data validity is first checked, and the reasonableness of parameter values ​​is judged to ensure that abnormal or erroneous data does not enter the subsequent processing flow. Then, the data is standardized in format, and the field naming, data type and unit are unified to meet the requirements of the internal business model. At the same time, metadata enhancement is performed on the data, including generating a unique serial number, adding a processing timestamp and source identifier, thereby ensuring data traceability and providing accurate and reliable input for subsequent batch processing. After being processed by the thread pool, the output of each task is a standardized data object, which meets the requirements of subsequent processing stages in terms of content integrity and format standardization.

[0022] The present invention is further configured such that S3 includes: The routing decision-maker performs multi-dimensional evaluation of each standardized data object and generates dynamic routing weights for each data object. The multi-dimensional assessment includes data value assessment based on predefined parameter importance mapping, assessment by comparing the degree of change between current data and historical cached data, and system load assessment by real-time monitoring of system resource indicators; The dynamic routing weights are compared with the preset priority threshold range, and the data objects are divided into three levels: high priority, normal priority and low priority based on the comparison results. Based on the priority ranking, a corresponding priority tag is attached to each data object. Specifically, the independent routing decision-maker component employs a multi-dimensional evaluation mechanism to intelligently analyze each standardized data object and generate routing decisions. In the data value assessment phase, a predefined parameter importance mapping table is used to determine the data value weight. This mapping table is pre-defined according to business needs, classifying different device types and parameter types. For example, key performance parameters such as CPU utilization and memory utilization of core servers are assigned the highest weight, while auxiliary parameters such as log levels of ordinary devices are assigned lower weights. This hierarchical mechanism ensures that critical business data receives priority processing. In the data change rate assessment phase, historical values ​​of the same parameter for the same device are retrieved from the cache database. The data change rate is determined by calculating the difference between the current value and the historical value; the greater the difference, the higher the change rate score. For parameters reported for the first time, a higher change rate score is assigned by default to ensure timely processing of new data. In the system load assessment phase, the routing decision-maker collects two sets of key indicators in real time: thread pool load indicators and queue load indicators. The thread pool load metric is the ratio of the current active threads to the maximum number of threads, and the queue load metric is the ratio of the current queue data volume to the queue capacity. A weighted algorithm is used to comprehensively calculate the real-time system load, with a higher weight for the thread pool load and a relatively lower weight for the queue load. This design is because the thread pool load directly affects the system's processing capacity. During the decision logic execution phase, the routing decision-maker weights the results of the three evaluation dimensions to generate dynamic routing weights. These dynamic routing weights are compared with preset priority threshold ranges: when the dynamic routing weight is greater than the first priority threshold, it is marked as high priority; when the dynamic routing weight is greater than the second priority threshold and less than or equal to the first priority threshold, it is marked as normal priority; when the dynamic routing weight is less than or equal to the second priority threshold, it is marked as low priority. This three-level classification mechanism ensures fine-grained management of data routing. The entire routing decision process establishes a decision caching mechanism, caching decision results for data of the same type to further improve processing efficiency. Each data object is assigned a corresponding priority tag after evaluation, providing a basis for subsequent differentiated processing.

[0023] The present invention is further configured such that S4 includes: A differentiated data routing strategy is executed based on the priority tag attached to the data object, and the routing strategy is processed according to three priority levels respectively; For data objects marked as high priority, update operations in the cache database and add operations in the batch write queue are performed simultaneously. The cache update adopts a predetermined data structure with the device identifier as the primary key and the combination of unit identifier and parameter name as fields. For data objects marked as normal priority, a single routing operation is performed, directly adding the data object to the end of the batch write queue without performing an update operation on the cache database; For data objects marked as low priority, a data discard operation is performed, terminating the subsequent processing flow of that data object. Specifically, the priority tag attached to each data object is first read, and the corresponding data routing strategy is executed according to different levels. For high-priority data objects, a dual-path parallel processing mechanism is initiated: in the cache update path, a specific data structure is used for storage: the device's unique identifier is used as the primary key, and a corresponding storage unit is established in the cache database; the combination of the device unit number and the performance parameter name is used as the field identifier, and the current data value and a timestamp accurate to milliseconds are used as the field value for storage; this hierarchical storage structure ensures that data from different units and different parameters under the same device can be stored independently and retrieved quickly. In the queue write path, data objects are simultaneously added to the batch. At the end of the write queue, data awaits subsequent batch processing. For ordinary priority data objects, a single-path processing strategy is adopted: the data object is directly added to the batch write queue, but the cache update operation is skipped. This processing method reduces the write pressure on the cache database while ensuring data persistence, and is particularly suitable for business data with high system load or low real-time requirements. For low priority data objects, a data discard operation is performed: before discarding, an operation log can be selectively recorded, including basic information of the data object, the reason for discarding, and timestamps, for subsequent auditing and analysis. After that, the processing flow of the data object is terminated, and system resources will be concentrated on processing higher priority data. The entire processing is managed through a state machine pattern to ensure that each data object can flow accurately according to the predetermined routing strategy.

[0024] The present invention is further configured such that S5 includes: The status changes of the batch queue are continuously monitored through a separately configured daemon thread or a timed task. Set dual trigger conditions for batch processing. The first trigger condition is that the backlog of ordinary priority data objects in the queue reaches a preset threshold. The second trigger condition is that the time interval since the last batch processing operation reaches a preset maximum time window. A first-come, first-served trigger mechanism is adopted, and batch processing operations are started immediately when any trigger condition is met first. During batch processing, data objects are extracted from the queue in descending order of their priority, with high-priority data objects being processed first and ordinary-priority data objects being processed subsequently. The extracted data objects are combined to form a batch data set. Specifically, a scheduled task mechanism is used to scan the status indicators of the batch queue at a fixed frequency, such as 4 times per second. The status indicators include the backlog of data objects of each priority, the total length of the queue, and the timestamp of the most recent batch processing. A trigger condition maintenance mechanism is set, including two dynamically adjustable trigger thresholds: one is a quantity threshold, which is automatically adjusted according to the real-time system load. For example, the base value is preset to 1000 ordinary priority data objects, but this threshold can be appropriately reduced to speed up the processing frequency when the system is under high load. The other is a time threshold, such as a preset maximum waiting window of 2 seconds, to ensure timely processing even when the data volume is small and to prevent data from being stuck for a long time. When any trigger condition is detected, the batch processing process is started immediately. The processing follows the principle of priority first: first, all high-priority data objects are scanned and extracted from the head of the queue to ensure that critical data is processed in a timely manner; then, ordinary priority data objects are extracted. Prioritizing data objects until the maximum capacity limit of this batch processing is reached, the number of data objects extracted each time and their priority distribution are recorded for subsequent performance analysis and parameter optimization. During data extraction, transactional operations are used to ensure data consistency. First, the queue is locked, then data is extracted in priority order, and finally the lock is released and batch processing begins. This mechanism avoids data inconsistency caused by the addition of new data during extraction. After extraction, data objects of different priorities are merged into a batch data set, and a unique batch identifier is generated for this set. At the same time, the batch's metadata information, including batch size, priority distribution, processing timestamp, etc., is recorded. This information will be passed to subsequent processing stages along with the batch. The entire batch processing process also includes a comprehensive exception handling mechanism. If an exception occurs during data extraction or set construction, a detailed error log is recorded, and a decision is made on whether to retry or skip the current batch based on the exception type.

[0025] The present invention is further configured such that S6 includes: Convert batch datasets into batch operation commands optimized for columnar databases; The entire batch of data can be submitted to a columnar database for storage through a single database interaction session. Leveraging the batch write capabilities of columnar databases, asynchronous non-blocking methods are employed to complete data persistence operations. Specifically, the received batch data set is first restructured into a columnar structure, categorizing and grouping data according to field types to form a memory layout conforming to the storage characteristics of columnar databases. Subsequently, a binary encoding protocol is used to convert the data set into a compact transmission format, describing the field structure through a metadata header, while the data body uses contiguous storage to optimize I / O efficiency. A reusable database connection session is established, completing the transmission of the entire batch of data through a single network interaction. A frame transmission mechanism is used at the network layer to ensure reliable delivery of large data packets. An asynchronous non-blocking write mode is enabled, where the main thread submits the data packet to the write queue and returns immediately, with a dedicated worker thread performing the actual database insertion operation and asynchronously processing the write results through a callback mechanism. An intelligent error recovery mechanism is also integrated, employing an exponential backoff retry strategy for network anomalies and automatically correcting and isolating data format errors. The system also monitors the database load status in real time, dynamically adjusting the batch submission scale to maximize throughput performance while ensuring data reliability. Finally, the batch loading interface of the columnar storage engine completes the efficient data persistence to disk.

[0026] The present invention is further configured such that S7 includes: Establish a unified query interface at the application service layer to receive external query requests; Perform feature analysis on the received query requests and determine the route based on whether the query conditions contain time range parameters or aggregation operation identifiers; For real-time query requests, they are routed to the cache database, and the latest values ​​of the specified device parameters are retrieved using key-value retrieval. For historical query requests, they are routed to a columnar database, where complex queries and data aggregation operations are performed using structured query statements. Specifically, by analyzing the spatiotemporal characteristics of query requests, different types of queries are intelligently routed to the most suitable storage engine, thereby optimizing query performance. This mechanism first establishes a unified query access point at the application layer, with all query requests entering the system through a standardized interface. The interface integrates a request feature analysis engine, which determines the query type by parsing key identifiers in the query parameters, such as time range and real-time requirements. For real-time queries that only require the latest data status, the system identifies their low latency and single-record access characteristics and routes them to a memory-based key-value store database. This type of database directly uses hash indexes to define the storage engine. The system returns results within microseconds, perfectly matching the needs of real-time monitoring scenarios. For historical queries involving historical time ranges, multi-dimensional aggregation, or complex analysis, the system routes them to a columnar storage database based on their characteristics of batch data scanning and computational intensity. The columnar database, through optimization techniques such as column-based storage and vectorized execution, can quickly scan massive amounts of historical data, making it particularly suitable for aggregation analysis operations. The system automatically converts query conditions into optimized SQL statements and returns results using the parallel processing capabilities of the columnar database. To achieve seamless routing, the system maintains a real-time data state mapping table to ensure data consistency between the two storage engines. All real-time data updates are synchronized to the in-memory database, while batch archived data is periodically persisted to the columnar database.

[0027] The invention further includes a fault tolerance mechanism in the method. When batch data persistence fails, the unsuccessful data objects are temporarily stored in a buffer and retried with priority in the next batch processing cycle. Specifically, when the batch data persistence operation fails, the system immediately triggers an exception capture process. First, it records a detailed error log and identifies the type of failure, such as network interruption, database overload, or abnormal data format. Then, it completely stores all data objects of the failed batch in a dedicated disk buffer and marks the batch with a failure status, timestamp, and retry count. The system maintains a retry queue priority mechanism, and the temporarily stored data will be retrieved and processed with priority in the next batch processing cycle. An exponential backoff strategy is used to dynamically adjust the retry interval, such as a 2-second interval for the first retry and doubling it for each subsequent retry. If the retry still fails after exceeding a preset threshold, a degradation scheme is initiated, and the data is transferred to a backup storage system. At the same time, an alarm is sent to notify the operation and maintenance personnel to intervene. Throughout the process, the system ensures that data is not lost or duplicated during buffering and retrying through transaction guarantees and monitors the retry success rate and system recovery status in real time until all data is successfully persisted.

[0028] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A thread pool and cache collaborative scheduling method for instantaneous high-concurrency scenarios, characterized in that, include: S1: Asynchronously receive performance data streams reported by external systems through message middleware, and deserialize and decode the received data messages to obtain structured data objects; S2: Use a parameterized thread pool to schedule and process structured data objects, and standardize the data objects to obtain standardized data objects; S3: Perform multi-dimensional evaluation on each standardized data object, generate dynamic routing weights, and assign corresponding priority tags to the data objects based on the comparison results of the dynamic routing weights and preset thresholds; S4: Perform differentiated data routing based on priority markers, write high-priority data objects to the cache and store them in the batch queue, store ordinary-priority data objects only in the batch queue, and discard low-priority data objects. S5: Monitor the status of the batch queue and trigger batch processing operations based on the dual conditions of queue data object backlog and maximum waiting time; S6: The batch data set retrieved from the batch queue after the trigger condition is met is asynchronously persisted to the columnar database in a single operation; S7: Identifies and routes received data query requests. Real-time query requests are directed to the cache to obtain the latest data, while query requests involving historical ranges or aggregation operations are directed to the columnar database for processing.

2. The thread pool and cache collaborative scheduling method for instantaneous high-concurrency scenarios according to claim 1, characterized in that, S1 includes: Configure the message middleware consumer client in the data processing service so that it can establish a connection with the specified message topic through a subscription mechanism; The consumer client continuously listens to the message topic through polling. When it detects that a new performance data message has been published, it retrieves the message content in an asynchronous and non-blocking manner. The obtained message body is parsed, and the message body is encoded using a predefined serialization format; Based on a predefined data schema specification, the serialized byte stream is converted into a structured data object that can be operated on internally by the service through deserialization operations.

3. The thread pool and cache collaborative scheduling method for instantaneous high-concurrency scenarios according to claim 1, characterized in that, S2 includes: Configure the thread pool executor, which includes a preset number of core threads, a maximum number of threads, and a bounded blocking queue with a fixed capacity; The decoded structured data object is encapsulated into an executable task, and the executable task is submitted to the thread pool executor through the submission method; After a task is submitted, if the core thread limit has not been reached, a new thread is created to process it. If the core thread limit has been reached, the task is stored in a bounded blocking queue for buffering. When the bounded blocking queue reaches its capacity limit, a new thread is started until the maximum number of threads is reached. When the number of threads reaches the maximum value and the queue is full, subsequent submitted tasks are processed according to the preset rejection policy. The tasks executed in the thread pool include the processing logic of data objects, including data validity verification, data format standardization processing, and supplementation of metadata information; The thread pool processes the data, transforming it from structured data objects to standardized data objects.

4. The thread pool and cache collaborative scheduling method for instantaneous high-concurrency scenarios according to claim 1, characterized in that, S3 includes: The routing decision-maker performs multi-dimensional evaluation of each standardized data object and generates dynamic routing weights for each data object. The multi-dimensional assessment includes data value assessment based on predefined parameter importance mapping, assessment by comparing the degree of change between current data and historical cached data, and system load assessment by real-time monitoring of system resource indicators; The dynamic routing weights are compared with the preset priority threshold range, and the data objects are divided into three levels: high priority, normal priority and low priority based on the comparison results. Based on the priority level classification, a corresponding priority tag is attached to each data object.

5. The thread pool and cache collaborative scheduling method for instantaneous high-concurrency scenarios according to claim 1, characterized in that, S4 includes: A differentiated data routing strategy is executed based on the priority tag attached to the data object, and the routing strategy is processed according to three priority levels respectively; For data objects marked as high priority, update operations in the cache database and add operations in the batch write queue are performed simultaneously. The cache update adopts a predetermined data structure with the device identifier as the primary key and the combination of unit identifier and parameter name as fields. For data objects marked as normal priority, a single routing operation is performed, directly adding the data object to the end of the batch write queue without performing an update operation on the cache database; For data objects marked as low priority, perform a data discard operation to terminate the subsequent processing flow of that data object.

6. The thread pool and cache collaborative scheduling method for instantaneous high-concurrency scenarios according to claim 1, characterized in that, S5 includes: The status changes of the batch queue are continuously monitored through a separately configured daemon thread or a timed task. Set dual trigger conditions for batch processing. The first trigger condition is that the backlog of ordinary priority data objects in the queue reaches a preset threshold. The second trigger condition is that the time interval since the last batch processing operation reaches a preset maximum time window. A first-come, first-served trigger mechanism is adopted, and batch processing operations are started immediately when any trigger condition is met first. During batch processing, data objects are extracted from the queue in descending order of their priority, with high-priority data objects being processed first and ordinary-priority data objects being processed subsequently. The extracted data objects are combined to form a batch data set.

7. The thread pool and cache collaborative scheduling method for instantaneous high-concurrency scenarios according to claim 1, characterized in that, S6 includes: Convert batch datasets into batch operation commands optimized for columnar databases; The entire batch of data can be submitted to a columnar database for storage through a single database interaction session. By leveraging the batch write capabilities of columnar databases, asynchronous non-blocking methods are used to complete data persistence operations.

8. The thread pool and cache collaborative scheduling method for instantaneous high-concurrency scenarios according to claim 1, characterized in that, S7 includes: Establish a unified query interface at the application service layer to receive external query requests; Perform feature analysis on the received query requests and determine the route based on whether the query conditions contain time range parameters or aggregation operation identifiers; For real-time query requests, they are routed to the cache database, and the latest values ​​of the specified device parameters are retrieved using key-value retrieval. For historical query requests, they are routed to a columnar database, where complex queries and data aggregation operations are performed using structured query statements.

9. The thread pool and cache collaborative scheduling method for instantaneous high-concurrency scenarios according to claim 1, characterized in that, The method also includes a fault tolerance mechanism, which temporarily stores the unsuccessful data objects in a buffer and retry them in the next batch processing cycle when batch data persistence fails.

Images