Multi-thread single-task asynchronous export method for large-scale data
Through dynamic thread pool routing and elegant downtime mechanism, the problem of unadjustable task dynamic routing and thread pool parameters in multi-threaded single-task asynchronous processing mode is solved, efficient, stable and secure data processing is achieved, and system response speed and resources are improved. Management efficiency.
Patent Information
- Application Number
- CN202411071642.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing multi-threaded single-task asynchronous processing mode lacks dynamic task routing, thread pool parameters cannot be dynamically adjusted, and elegant shutdown functions are missing, resulting in low system response speed, insufficient resource management and reduced task execution quality, affecting system stability and security.
The dynamic thread pool routing module is used to allocate dedicated thread pools to different categories of tasks, supports dynamic configuration of thread pool parameters, realizes dynamic routing and resource isolation of tasks, combines elegant downtime mechanisms and cloud storage modules to ensure that tasks are allocated according to priority and resource requirements. Provides efficient, stable and secure data processing.
It significantly improves the efficiency of large-scale data processing and system response speed, ensures that tasks are executed according to priority, avoid resource interference, provide high availability and security, and simplifies development, operation and maintenance processes.
Smart Images

Figure CN119065807A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of data export, in particular to a multi-threaded single-task asynchronous export method for large-scale data. Background Art
[0002] In the field of modern software development and big data processing, efficiency and response speed are key indicators for measuring system performance. Among them, "multi-threaded single-task asynchronous processing" as an efficient data processing mode is widely used in scenarios such as file export and data capture, especially in processing large-scale data export, which can significantly improve processing speed and system responsiveness.
[0003] In modern software development, especially when dealing with big data and complex business logic, efficiency and response speed are directly related to user experience and system performance. High efficiency means that the system can process requests and data faster, reducing user waiting time. Fast response means that the system can respond to user operations or external events instantly, providing a smooth user interaction experience.
[0004] "Multi-threaded single-task asynchronous processing" is a strategy that splits a single complex task into multiple threads for parallel execution, while using asynchronous methods to handle IO waiting or time-consuming operations. Through multi-threading, the system can fully utilize the computing power of multi-core processors to achieve concurrent execution of tasks, thereby improving the overall processing speed; asynchronous processing allows threads to continue to execute other tasks while waiting for IO operations (such as disk reading and writing, network requests, etc.) to complete, reducing thread idle time and improving resource utilization.
[0005] Multithreading means running multiple threads simultaneously in a program, and each thread can independently execute different parts of the program code. Multithreading allows the program to process multiple tasks in parallel, thereby improving the execution efficiency and response speed of the program. When processing large amounts of data and exporting them to files, using multithreaded single-task asynchronous processing can significantly speed up the export, especially when the file is large or there is a lot of data.
[0006] Asynchronous processing means that when a program performs an operation, it will not wait for the operation to complete before continuing to execute subsequent code. When the asynchronous operation is completed, the program will process the operation result through some mechanism (such as callback function, event, etc.). When crawling data from the Internet or other data sources, the crawling process may be slow due to factors such as network latency and large data volume. Through multi-threaded asynchronous processing, multiple data sources or data fragments can be crawled in parallel, thereby speeding up the crawling speed.
[0007] The existing multi-threaded single-task asynchronous processing mode has the following shortcomings:
[0008] Dynamic routing of tasks is not implemented, which makes it impossible to automatically assign tasks to the most appropriate thread pool for execution based on their type, priority, and resource requirements.
[0009] Thread pool parameters cannot be modified dynamically, which will reduce the flexibility and response speed of the system in practical applications;
[0010] The graceful shutdown function is not implemented, resulting in the thread pool being unable to gradually stop accepting new tasks when receiving a shutdown signal, affecting the stability and security of the system;
[0011] Different types of tasks interfere with each other during execution, which may lead to a decline in the execution quality of key tasks and affect the overall stability of the system.
[0012] Therefore, it is necessary to solve the above-mentioned problem of insufficient performance optimization.
[0013] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to those skilled in the art. Summary of the invention
[0014] In view of the defects existing in the prior art, the purpose of the present invention is to provide a multi-threaded single-task asynchronous export method for large-scale data, which effectively solves the problems of low data processing efficiency, insufficient resource management, lack of dynamic task scheduling, etc. existing in the prior art, and provides an efficient, stable and secure data processing solution for modern software development and big data processing fields.
[0015] In order to achieve the above purpose, the technical solution adopted by the present invention is:
[0016] A multi-threaded single-task asynchronous export method for large-scale data, characterized by comprising:
[0017] Step 1: Call the configuration rule module, set the rules and generate the export configuration template;
[0018] Step 2: Call the dynamic thread pool routing module and use a concurrent processing mechanism to allocate dedicated thread pools for different types of tasks to optimize resource management and improve system throughput; each thread pool performs data export according to its corresponding export configuration template;
[0019] The dynamic thread pool routing module supports dynamic adjustment of the core parameters of the thread pool, realizes dynamic routing of tasks, and ensures that each task is assigned to the most appropriate thread pool for execution according to its type, priority and resource requirements;
[0020] The core parameters are any one, part or all of the following: number of core threads, maximum number of threads, queue strategy; the core parameters of the thread pool can be dynamically configured and take effect in real time without restarting the service;
[0021] Step 3: Call the resource isolation module to isolate the thread pool to ensure that different types of tasks do not interfere with each other during execution, thereby improving the execution quality of key tasks and the stability of the system;
[0022] The system implements resource isolation between tasks by allocating an independent execution environment and resources to each thread pool. During task execution, the system ensures that tasks in different thread pools do not interfere with each other, thereby improving the execution quality of key tasks and the stability of the system.
[0023] Based on the above technical solution, in step 1, the setting rules specifically include:
[0024] First, the integrated system is selected as the data source of the data to be exported, and the data request address of the integrated system is obtained by selecting;
[0025] Then, a module to be exported in the integrated system is selected; the integrated system includes several modules, and the module name and module code are obtained by selecting the module to be exported;
[0026] Then set the upper limit of the number of data items for each data request;
[0027] Finally, set the thread waiting time.
[0028] Based on the above technical solution, in step 2, the queue strategy determines how new tasks should be processed when all threads in the thread pool are busy, including but not limited to the following queue strategies:
[0029] Infinite queue: New tasks will be queued indefinitely for execution until there is an idle thread available in the thread pool;
[0030] Bounded queue: The queue size is limited. When the queue is full, new tasks will be processed according to a specific rejection policy.
[0031] Blocking queue: When the queue is full, the submission of new tasks will be blocked until space is available in the queue.
[0032] On the basis of the above technical solution, in step 3, the system also monitors and limits the resource usage of the thread pool to prevent a single task from occupying too many resources and affecting the execution of other tasks;
[0033] The monitoring and limiting include setting the maximum number of concurrent connections and limiting memory usage.
[0034] On the basis of the above technical solutions, it also includes:
[0035] Step 4: Call the performance optimization module to reduce thread context switching according to the thread pool routing strategy, improve the CPU cache hit rate, and thus improve the overall performance of the system.
[0036] Based on the above technical solution, in step 4, when the data export task is triggered, the performance optimization module first analyzes the parameters of the task, and then dynamically allocates the task to the most appropriate thread pool according to the predefined routing strategy;
[0037] The parameters of the task include:
[0038] Task type, which classifies tasks into different thread pools according to their nature;
[0039] Priority, allocating more resources or shorter waiting time to high-priority tasks;
[0040] Resource requirements: Consider the task's requirements for CPU, memory and other resources, and distribute them to thread pools with sufficient resources;
[0041] Execution time, special handling is performed on tasks that require a long execution time, such as assigning them to a separate thread pool to avoid blocking other tasks.
[0042] Based on the above technical solution, during the task execution process, the status and performance indicators of the thread pool are monitored in real time, and the parameters of the thread pool are dynamically adjusted according to the actual situation to optimize resource allocation and improve system throughput;
[0043] The state and performance indicators of the thread pool include: workload and response time of the thread pool;
[0044] The parameters of the thread pool are dynamically adjusted, including: increasing or decreasing the number of core threads, increasing or decreasing the maximum number of threads, and adjusting the queue strategy.
[0045] Based on the above technical solution, during the execution of the data export task, when the system receives a shutdown signal, the elegant shutdown module is called to first stop receiving new tasks, then process the tasks in the queue one by one, and finally close the thread pool to ensure a smooth transition of the system.
[0046] Based on the above technical solutions, during the execution of the data export task, the cloud storage module is called to store the exported files in the cloud, and the high availability, scalability, high security, and high maintainability of cloud services are utilized to achieve fast data access, redundant backup, and simplified development and operation processes.
[0047] Based on the above technical solution, the cloud storage service will automatically process the data as follows:
[0048] Data sharding and replication: Data is sharded and stored in multiple different physical locations, and each shard has multiple copies;
[0049] Cross-region replication: For data that needs to be accessed globally, cloud service providers may also replicate it to data centers in different regions to provide wider access coverage and higher fault tolerance.
[0050] Regular backup and snapshots: In addition to real-time replication, cloud services also regularly back up and snapshot stored data to prepare for possible catastrophic events;
[0051] Cloud storage services also take the following multi-level security measures:
[0052] Data encryption: Encryption technology is used during data transmission and storage to ensure the confidentiality and integrity of data;
[0053] Access control: Through strict identity authentication and permission management mechanisms, control user access rights to data to prevent unauthorized access and data leakage;
[0054] Firewall and security audit: Deploy firewalls and other network security devices to monitor and prevent potential attacks; record user operations and security incidents for post-audit and tracking;
[0055] Compliance certification: Cloud service providers will comply with relevant data protection regulations and standards and pass corresponding compliance certification to ensure the security and compliance of services.
[0056] The multi-threaded single-task asynchronous export method for large-scale data described in the present invention has the following beneficial effects:
[0057] 1. By introducing dynamic thread pool routing and refined thread management strategies, the efficiency of large-scale data processing and system response speed are significantly improved. This method allows the system to dynamically assign tasks to the most appropriate thread pool according to the type, priority and resource requirements of the task, optimizing the resource allocation and task scheduling process. In addition, the dynamic thread pool configuration function allows system administrators to adjust thread pool parameters in real time without restarting the service, further improving the flexibility and responsiveness of the system;
[0058] 2. The graceful shutdown mechanism ensures that when a shutdown signal is received, the thread pool can safely and orderly stop receiving new tasks and complete the tasks in the queue, ensuring the stability of the system and the consistency of data. The resource isolation strategy ensures that tasks do not interfere with each other through thread pool isolation, improving the execution quality of key tasks and the stability of the overall system;
[0059] 3. Performance optimization measures have been taken to reduce thread context switching and improve CPU cache hit rate, further improving the overall performance of the system; the integration of cloud storage not only provides a high-availability, scalable and highly secure data storage solution, but also accelerates global access speed through distributed CDN services, simplifies development and operation and maintenance processes, and ensures data security and compliance.
[0060] Through the above-mentioned refined thread management strategy, not only can high-concurrency requests be handled effectively, but also important tasks can be given priority, while maintaining the rational use of resources and the long-term stable operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] The present invention has the following accompanying drawings:
[0062] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.
[0063] Figure 1 A flowchart of Embodiment 1 of a multi-threaded single-task asynchronous export method for large-scale data described in the present invention. DETAILED DESCRIPTION
[0064] The present invention is further described in detail below in conjunction with the accompanying drawings. The detailed description is an explanation made in conjunction with the exemplary embodiments of the present invention, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0065] like Figure 1 As shown, the present invention provides a multi-threaded single-task asynchronous export method for large-scale data, comprising:
[0066] Step 1: Call the configuration rule module, set the rules and generate the export configuration template;
[0067] Exemplarily, the setting rules specifically include:
[0068] First, the integrated system is selected as the data source of the data to be exported, and the data request address of the integrated system is obtained by selecting;
[0069] Then, a module to be exported in the integrated system is selected; the integrated system includes several modules, and the module name and module code are obtained by selecting the module to be exported;
[0070] Then set the upper limit of the number of data items for each data request;
[0071] Finally, set the thread waiting time;
[0072] Step 2: Call the dynamic thread pool routing module and use a concurrent processing mechanism to allocate dedicated thread pools for different types of tasks to optimize resource management and improve system throughput; each thread pool performs data export according to its corresponding export configuration template;
[0073] The dynamic thread pool routing module supports dynamic adjustment of the core parameters of the thread pool, realizes dynamic routing of tasks, and ensures that each task is assigned to the most appropriate thread pool for execution according to its type, priority and resource requirements;
[0074] The core parameters are any one, part or all of the following: number of core threads, maximum number of threads, queue strategy; the core parameters of the thread pool can be dynamically configured and take effect in real time without restarting the service; among them, the queue strategy determines how new tasks should be handled when all threads in the thread pool are busy (such as putting them in the queue for waiting, rejecting new tasks, etc.);
[0075] In the dynamic thread pool configuration, the queue strategy is an important parameter, which determines how the newly arrived tasks should be handled when all threads in the thread pool are busy; the system of the present invention supports dynamic configuration of the queue strategy, including but not limited to the following queue strategies:
[0076] Infinite queue: New tasks will be queued indefinitely for execution until there is an idle thread available in the thread pool. Although this method will not cause tasks to be rejected, it may consume a lot of memory resources due to the infinite growth of the queue.
[0077] Bounded queue: The queue size is limited. When the queue is full, new tasks will be processed according to a specific rejection strategy (such as throwing an exception, calling a rejection handler, etc.); this method can effectively prevent memory resource exhaustion;
[0078] Blocking queue: When the queue is full, the submission of new tasks will be blocked until there is space in the queue. This method can control the submission rate of tasks to a certain extent and avoid system overload.
[0079] Users can dynamically adjust the queue strategy through the system interface or configuration files according to actual needs to adapt to different application scenarios and load conditions;
[0080] Step 3: Call the resource isolation module to isolate the thread pool to ensure that different types of tasks do not interfere with each other during execution, thereby improving the execution quality of key tasks and the stability of the system;
[0081] The system implements resource isolation between tasks by allocating an independent execution environment and resources (such as CPU, memory, etc.) to each thread pool. During task execution, the system ensures that tasks in different thread pools do not interfere with each other, thereby improving the execution quality of key tasks and the stability of the system. In addition, the system also monitors and limits the resource usage of the thread pool (such as setting the maximum number of concurrency, limiting memory usage, etc.) to prevent a single task from occupying too many resources and affecting the execution of other tasks.
[0082] On the basis of the above technical solutions, it also includes:
[0083] Step 4: Call the performance optimization module to reduce thread context switching according to the thread pool routing strategy, improve the CPU cache hit rate, and thus improve the overall performance of the system;
[0084] Specifically include:
[0085] When a data export task is triggered, the performance optimization module first analyzes the task parameters and then dynamically allocates the task to the most appropriate thread pool based on the predefined routing strategy;
[0086] The parameters of the task include:
[0087] Task type, which classifies tasks into different thread pools according to their nature (such as data processing, IO operation, etc.);
[0088] Priority, allocating more resources or shorter waiting time to high-priority tasks;
[0089] Resource requirements: Consider the task's requirements for CPU, memory and other resources, and distribute them to thread pools with sufficient resources;
[0090] Execution time: Special handling is performed on tasks that take a long time to execute, such as assigning them to separate thread pools to avoid blocking other tasks.
[0091] During the task execution process, the thread pool status and performance indicators are monitored in real time, and the thread pool parameters are dynamically adjusted according to the actual situation to optimize resource allocation and improve system throughput;
[0092] The state and performance indicators of the thread pool include: workload and response time of the thread pool;
[0093] The parameters of the thread pool are dynamically adjusted, including: increasing or decreasing the number of core threads, increasing or decreasing the maximum number of threads, and adjusting the queue strategy.
[0094] During task scheduling, the system tries to assign tasks of the same or similar types to the same thread pool to reduce the number of thread context switches. At the same time, the system also improves the CPU cache hit rate and reduces the frequency and latency of CPU access to memory by optimizing data structures, algorithms, and cache strategies. These measures work together to significantly improve the overall performance of the system.
[0095] Exemplarily, the improving the CPU cache hit rate includes the following measures:
[0096] Data locality optimization: By reasonably arranging the data access order and storage structure (such as using the locality principle to optimize the data structure layout), the number and distance of CPU access to memory can be reduced, and the cache hit rate can be improved;
[0097] Cache prefetching: Loading data from memory into cache before the CPU actually needs the data to reduce cache misses;
[0098] Code optimization: write efficient code to avoid repeated access to discontinuous memory addresses in loops and reduce cache pollution and conflicts;
[0099] These measures work together to effectively improve the CPU cache hit rate, thereby accelerating program execution and improving system performance.
[0100] Based on the above technical solution, during the execution of the data export task, when the system receives a shutdown signal, the elegant shutdown module is called to first stop receiving new tasks, then process the tasks in the queue one by one, and finally close the thread pool to ensure a smooth transition of the system.
[0101] When the system receives a shutdown signal, the graceful shutdown module first marks the system as "shutdown" and stops receiving new task requests. Then, the module traverses all thread pools and checks whether the tasks in the queue have been completed one by one. For tasks that are still being executed, the system gives a certain grace period (such as the default 30 seconds) to allow them to continue to execute until they are completed. After the grace period, for tasks that are still unfinished, the system can choose to record them in the log for subsequent processing, or directly terminate these tasks. Finally, after all tasks are processed, the system closes all thread pools and releases related resources to complete the graceful shutdown process.
[0102] Based on the above technical solutions, during the execution of the data export task, the cloud storage module is called to store the exported files in the cloud, and the high availability, scalability, high security, and high maintainability of cloud services are utilized to achieve fast data access, redundant backup, and simplified development and operation processes.
[0103] When the data export task is completed, the system uploads the exported files to the cloud storage service. Before uploading, the system will perform necessary processing on the files (such as compression, encryption, etc.) to reduce the file size and improve data security. Then, the system uses the API interface provided by the cloud storage service to upload the files to the specified storage bucket or container. After the upload is completed, the system updates the file storage information in the database (such as file URL, upload time, etc.) for subsequent data access and management. At the same time, the system also uses the distributed CDN service of the cloud service to accelerate the global access speed of files and ensure the high availability, scalability and security of data.
[0104] For example, the cloud storage service will automatically process the data as follows:
[0105] Data sharding and replication: Data is sharded and stored in multiple different physical locations (such as different data centers or racks), and each shard has multiple copies;
[0106] Cross-region replication: For data that needs to be accessed globally, cloud service providers may also replicate it to data centers in different regions to provide wider access coverage and higher fault tolerance.
[0107] Regular backup and snapshots: In addition to real-time replication, cloud services also regularly back up and snapshot stored data to prepare for possible catastrophic events;
[0108] These measures ensure that even in the event of unexpected situations such as hardware failures, natural disasters, etc., user data can be protected and restored to the greatest extent possible.
[0109] For example, cloud storage services also take the following multi-level security measures:
[0110] Data encryption: Use encryption technology (such as SSL / TLS, AES, etc.) during data transmission and storage to ensure data confidentiality and integrity;
[0111] Access control: Through strict identity authentication and permission management mechanisms, control user access rights to data to prevent unauthorized access and data leakage;
[0112] Firewall and security audit: Deploy firewalls and other network security devices to monitor and prevent potential attacks; record user operations and security incidents for post-audit and tracking;
[0113] Compliance certification: Cloud service providers will comply with relevant data protection regulations and standards (such as GDPR, HIPAA, etc.) and pass corresponding compliance certifications (such as ISO 27001, SOC 2, etc.) to ensure the security and compliance of services;
[0114] These measures together constitute the security protection system of cloud services, providing users with reliable data protection services.
[0115] High availability and scalability: Cloud storage services are highly available, capable of automatic failover and load balancing, and support on-demand expansion of resources, ensuring service continuity and high data availability.
[0116] Accelerate global access: Using the distributed CDN service of cloud services, files can be transferred quickly, and users can obtain content from the server closest to the geographical location, greatly improving the access speed worldwide;
[0117] Data redundancy and backup: Cloud services automatically perform redundant backup of stored data, effectively preventing data loss and enhancing data security and reliability;
[0118] Simplify development and operation and maintenance: By uploading files to cloud services, the complexity of file uploading and management is simplified. Developers can focus on the development of core business logic without having to implement cumbersome file storage and management functions by themselves.
[0119] Security and compliance assurance: Cloud services provide multi-level security protection measures, including data encryption and access control, to help users meet relevant laws and regulations and standards for data protection.
[0120] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.
[0121] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by those skilled in the art based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.
Claims
1. A multi-threaded single-task asynchronous export method for large-scale data, characterized in that: include: Step 1: Call the configuration rule module, set the rules and generate the export configuration template; Step 2: Call the dynamic thread pool routing module and use the concurrent processing mechanism to allocate dedicated thread pools for different types of tasks to optimize resource management and improve system throughput; each thread pool performs data export according to its corresponding export configuration template; The dynamic thread pool routing module supports dynamic adjustment of the core parameters of the thread pool, realizes dynamic routing of tasks, and ensures that each task is assigned to the most appropriate thread pool for execution according to its type, priority and resource requirements; The core parameters are any one, part or all of the following: number of core threads, maximum number of threads, queue strategy; the core parameters of the thread pool can be dynamically configured and take effect in real time without restarting the service; Step 3: Call the resource isolation module to isolate the thread pool to ensure that different types of tasks do not interfere with each other during execution, thereby improving the execution quality of key tasks and the stability of the system; The system implements resource isolation between tasks by allocating an independent execution environment and resources to each thread pool. During task execution, the system ensures that tasks in different thread pools do not interfere with each other, thereby improving the execution quality of key tasks and the stability of the system.
2. A multi-threaded single-task asynchronous export method for large-scale data as claimed in claim 1, characterized in that: In step 1, the setting rules specifically include: First, the integrated system is selected as the data source of the data to be exported, and the data request address of the integrated system is obtained by selecting; Then, a module to be exported in the integrated system is selected; the integrated system includes several modules, and the module name and module code are obtained by selecting the module to be exported; Then set the upper limit of the number of data items for each data request; Finally, set the thread waiting time.
3. A multi-threaded single-task asynchronous export method for large-scale data as claimed in claim 1, characterized in that: In step 2, the queue strategy determines how new tasks should be handled when all threads in the thread pool are busy, including but not limited to the following queue strategies: Infinite queue: New tasks will be queued indefinitely for execution until there is an idle thread available in the thread pool; Bounded queue: The queue size is limited. When the queue is full, new tasks will be processed according to a specific rejection policy. Blocking queue: When the queue is full, the submission of new tasks will be blocked until space is available in the queue.
4. A multi-threaded single-task asynchronous export method for large-scale data as claimed in claim 1, characterized in that: In step 3, the system also monitors and limits the resource usage of the thread pool to prevent a single task from occupying too many resources and affecting the execution of other tasks; The monitoring and limiting include setting the maximum number of concurrent connections and limiting memory usage.
5. A multi-threaded single-task asynchronous export method for large-scale data as claimed in claim 1, characterized in that: Also includes: Step 4: Call the performance optimization module to reduce thread context switching according to the thread pool routing strategy, improve the CPU cache hit rate, and thus improve the overall performance of the system.
6. A multi-threaded single-task asynchronous export method for large-scale data as claimed in claim 5, characterized in that: In step 4, when the data export task is triggered, the performance optimization module first analyzes the task parameters and then dynamically allocates the task to the most appropriate thread pool according to the predefined routing strategy; The parameters of the task include: Task type, which classifies tasks into different thread pools according to their nature; Priority, allocating more resources or shorter waiting time to high-priority tasks; Resource requirements: Consider the task's requirements for CPU, memory and other resources, and distribute them to thread pools with sufficient resources; Execution time, special handling is performed on tasks that require a long execution time, such as assigning them to a separate thread pool to avoid blocking other tasks.
7. A multi-threaded single-task asynchronous export method for large-scale data as claimed in claim 6, characterized in that: During the task execution process, the thread pool status and performance indicators are monitored in real time, and the thread pool parameters are dynamically adjusted according to the actual situation to optimize resource allocation and improve system throughput; The state and performance indicators of the thread pool include: workload and response time of the thread pool; The parameters of the thread pool are dynamically adjusted, including: increasing or decreasing the number of core threads, increasing or decreasing the maximum number of threads, and adjusting the queue strategy.
8. A multi-threaded single-task asynchronous export method for large-scale data as claimed in claim 6, characterized in that: During the execution of the data export task, when the system receives a shutdown signal, the graceful shutdown module is called to first stop receiving new tasks, then process the tasks in the queue one by one, and finally close the thread pool to ensure a smooth transition of the system.
9. A multi-threaded single-task asynchronous export method for large-scale data as claimed in claim 6, characterized in that: During the execution of the data export task, the cloud storage module is called to store the exported files in the cloud, and the high availability, scalability, high security, and high maintainability of cloud services are utilized to achieve fast data access, redundant backup, and simplified development and operation processes.
10. A multi-threaded single-task asynchronous export method for large-scale data as claimed in claim 9, characterized in that: The cloud storage service will automatically process the data as follows: Data sharding and replication: Data is sharded and stored in multiple different physical locations, and each shard has multiple copies; Cross-region replication: For data that needs to be accessed globally, cloud service providers may also replicate it to data centers in different regions to provide wider access coverage and higher fault tolerance. Regular backup and snapshots: In addition to real-time replication, cloud services also regularly back up and snapshot stored data to prepare for possible catastrophic events; Cloud storage services also take the following multi-level security measures: Data encryption: Encryption technology is used during data transmission and storage to ensure the confidentiality and integrity of data; Access control: Through strict identity authentication and permission management mechanisms, control user access rights to data to prevent unauthorized access and data leakage; Firewall and security audit: Deploy firewalls and other network security devices to monitor and prevent potential attacks; record user operations and security incidents for post-audit and tracking; Compliance certification: Cloud service providers will comply with relevant data protection regulations and standards and pass corresponding compliance certification to ensure the security and compliance of services.
Citation Information
Cited By
General data exporting method and system, storage medium and electronic equipment
CN119988469A
A general data export method, system, storage medium and electronic device
CN119988469B
