A method for implementing multi-core concurrent load on cluster file system client

Abstract

The present invention relates to the technical field of load regulation, and in particular to a method for realizing multi-core concurrent load of a cluster file system client. The method comprises the following steps: obtaining cluster file client task data; performing task splitting on the cluster file client task data to generate subtask sorting data; performing available resource analysis on the cluster file client to generate client available resource data; performing virtualized ultra-fine-grained task division on the subtask sorting data according to the client available resource data to generate ultra-fine-grained virtual task list data; performing type judgment rule engine construction on the ultra-fine-grained virtual task list data to generate a task type judgment rule engine. The present invention improves the concurrent processing capability and overall performance of the system by performing task splitting, intelligent scheduling, asynchronous communication, and performance analysis on the cluster file client task data.

Description

Technical Field

[0001] The present invention relates to the technical field of load regulation, and in particular to a method for implementing multi-core concurrent load on a cluster file system client. Background Art

[0002] In the early stages of cluster file system development, most systems were single-core, and the communication and data transmission between the client and the server relied on the processing power of a single core. This led to performance bottlenecks, especially when facing large-scale data operations. With the development of cloud computing and big data, distributed file systems have become an important development direction for cluster file systems. Clients no longer communicate only with a single server, but can access data from multiple nodes at the same time. This requires clients to have stronger multi-core concurrent load capabilities to handle data read and write requests from multiple nodes. The rise of software-defined storage (SDS) technology has brought new possibilities for cluster file systems. By virtualizing and decoupling storage functions, cluster file system clients can adapt to different hardware environments more flexibly and better utilize the concurrent capabilities of multiple cores. However, the current existing technologies usually only consider the execution order of tasks for task scheduling, lack intelligent scheduling and optimization of task execution, and the communication between clients is usually synchronously blocked. After a client sends a request, it must wait for a response, which affects the concurrent processing capability, resulting in low concurrent processing capability and overall performance of the system. Summary of the invention

[0003] Based on this, it is necessary to provide a method for implementing multi-core concurrent load of a cluster file system client to solve at least one of the above technical problems.

[0004] To achieve the above object, a method for implementing multi-core concurrent load of a cluster file client is provided, the method comprising the following steps:

[0005] Step S1: Acquire cluster file client task data; perform task splitting on the cluster file client task data to generate subtask sorting data; perform available resource analysis on the cluster file client to generate client available resource data; perform virtualized ultra-fine-grained task division on the subtask sorting data according to the client available resource data to generate ultra-fine-grained virtual task list data;

[0006] Step S2: construct a type judgment rule engine for the ultra-fine-grained virtual task list data to generate a task type judgment rule engine; prioritize the ultra-fine-grained virtual task list data through the task type judgment rule engine to generate urgent task priority ranking data; perform intelligent cache task scheduling on the urgent task priority ranking data to generate multi-core concurrent task scheduling result data;

[0007] Step S3: performing high-order asynchronous message communication on the multi-core concurrent task scheduling result data to generate asynchronous message communication data; performing task decoupling on the asynchronous message communication data to generate asynchronous message decoupling data; performing non-blocking communication on the asynchronous message decoupling data to generate an asynchronous message communication queue;

[0008] Step S4: perform communication parallel pipeline data stream transmission on the asynchronous message communication queue to generate asynchronous parallel transmission information data; perform performance bottleneck analysis on the asynchronous parallel transmission information data to generate performance bottleneck analysis data; perform load regulation on the asynchronous parallel transmission information data through the performance bottleneck analysis data to generate multi-core concurrent load adjustment data.

[0009] The present invention can improve the parallelism of tasks by splitting and virtualizing tasks, so that multiple computing resources in the cluster can process different tasks at the same time, and speed up the overall processing speed. Through available resource analysis and ultra-fine-grained task division, the resources of each client in the cluster can be more fully utilized to avoid resource waste and bottlenecks. Meticulous task division and resource allocation help to improve the overall efficiency of the system, ensure that tasks can be completed in the shortest time, and better adapt to the heterogeneity of different clients in the cluster. Through the task type judgment rule engine and priority sorting, the priority of tasks can be effectively managed to ensure that urgent tasks are processed in time and improve the responsiveness of the system. Intelligent cache task scheduling can better utilize cluster resources, avoid resource waste, and improve the resource utilization and performance of the system. Multi-core concurrent task scheduling result data can guide the concurrent execution of tasks in a multi-core environment, improve the concurrent execution efficiency of the system, and accelerate the task processing speed. The use of high-order asynchronous message communication and task decoupling can improve the flexibility and performance of the system, making the system easier to expand and maintain. Through task decoupling, the dependency between the various modules of the system is reduced, and the maintainability and scalability of the system are improved. The use of non-blocking communication methods can improve the concurrency and responsiveness of the system, so that the system can handle a large number of task requests more effectively. Through asynchronous parallel transmission of information data generation, the system can utilize the ability of parallel processing to improve the efficiency and throughput of data transmission. Through performance bottleneck analysis and load regulation, the system can timely discover and solve performance bottleneck problems, thereby optimizing the performance of the system and improving the overall operating efficiency. The generated multi-core concurrent load adjustment data can help the system adjust the load in real time, and flexibly allocate resources according to the current operating status and performance requirements to ensure the stability and efficiency of the system under different workloads. Therefore, the present invention improves the concurrent processing capability and overall performance of the system by performing task splitting, intelligent scheduling, asynchronous communication, and performance analysis on cluster file client task data.

[0010] In this specification, a cluster file system is provided for executing the above-mentioned cluster file client multi-core concurrent load implementation method, and the cluster file system includes:

[0011] The task division module is used to obtain the cluster file client task data; perform task division on the cluster file client task data to generate subtask sorting data; perform available resource analysis on the cluster file client to generate client available resource data; perform virtualized ultra-fine-grained task division on the subtask sorting data according to the client available resource data to generate ultra-fine-grained virtual task list data;

[0012] The judgment engine module is used to construct a type judgment rule engine for the ultra-fine-grained virtual task list data and generate a task type judgment rule engine; prioritize the ultra-fine-grained virtual task list data through the task type judgment rule engine and generate urgent task priority ranking data; perform intelligent cache task scheduling on the urgent task priority ranking data and generate multi-core concurrent task scheduling result data;

[0013] The asynchronous communication module is used to perform high-order asynchronous message communication on the multi-core concurrent task scheduling result data to generate asynchronous message communication data; perform task decoupling on the asynchronous message communication data to generate asynchronous message decoupling data; perform non-blocking communication on the asynchronous message decoupling data to generate an asynchronous message communication queue;

[0014] The load regulation module is used to transmit the communication parallel pipeline data stream of the asynchronous message communication queue to generate asynchronous parallel transmission information data; perform performance bottleneck analysis on the asynchronous parallel transmission information data to generate performance bottleneck analysis data; perform load regulation on the asynchronous parallel transmission information data through the performance bottleneck analysis data to generate multi-core concurrent load adjustment data.

[0015] The beneficial effect of the present invention is that subsequent processing is prepared by obtaining task data of a file client from a cluster. Task data is split into smaller subtasks and they are sorted. This helps to better manage tasks and improve execution efficiency. Available resources of a cluster file client are analyzed, including computing resources, storage resources, etc. This helps to reasonably allocate and schedule subsequent tasks. According to the available resource data of the client, the subtasks are further divided to generate an ultra-fine-grained virtual task list. In this way, the execution of tasks and resource utilization can be more finely controlled. Based on the ultra-fine-grained virtual task list data, a rule engine is built to determine the task type. In this way, more flexible processing can be performed according to the characteristics and requirements of the task. Tasks are prioritized according to the urgency of the task and other factors, and intelligent cache task scheduling is performed. This helps to ensure that urgent tasks can be processed in a timely manner and improve the response speed and efficiency of the system. High-order asynchronous message communication is performed on the multi-core concurrent task scheduling result data to decouple the dependencies between tasks and improve concurrent processing capabilities. Asynchronous message communication data is converted into non-blocking communication, and an asynchronous message communication queue is generated. This helps to improve the concurrent processing capability and response speed of the system. The asynchronous message communication queue is communicated in parallel pipeline data stream transmission to further improve the communication efficiency and concurrent processing capability. The performance bottleneck analysis is performed on the asynchronous parallel transmission information data, and the data is load regulated according to the analysis results. This helps to optimize the performance of the system and improve the overall processing efficiency. Therefore, the present invention improves the concurrent processing capability and overall performance of the system by performing task splitting, intelligent scheduling, asynchronous communication, and performance analysis on the cluster file client task data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the steps of a method for implementing multi-core concurrent load on a cluster file client;

[0017] Figure 2 for Figure 1 Detailed implementation steps of step S2 in the flowchart;

[0018] Figure 3 for Figure 2 Detailed implementation steps of step S24;

[0019] Figure 4 for Figure 1 Detailed implementation steps of step S3 in FIG.

[0020] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] The following is a clear and complete description of the technical method of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0022] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0023] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0024] To achieve this, please refer to Figures 1 to 4 , a method for implementing multi-core concurrent load of a cluster file client, the method comprising the following steps:

[0025] Step S1: Acquire cluster file client task data; perform task splitting on the cluster file client task data to generate subtask sorting data; perform available resource analysis on the cluster file client to generate client available resource data; perform virtualized ultra-fine-grained task division on the subtask sorting data according to the client available resource data to generate ultra-fine-grained virtual task list data;

[0026] Step S2: construct a type judgment rule engine for the ultra-fine-grained virtual task list data to generate a task type judgment rule engine; prioritize the ultra-fine-grained virtual task list data through the task type judgment rule engine to generate urgent task priority ranking data; perform intelligent cache task scheduling on the urgent task priority ranking data to generate multi-core concurrent task scheduling result data;

[0027] Step S3: performing high-order asynchronous message communication on the multi-core concurrent task scheduling result data to generate asynchronous message communication data; performing task decoupling on the asynchronous message communication data to generate asynchronous message decoupling data; performing non-blocking communication on the asynchronous message decoupling data to generate an asynchronous message communication queue;

[0028] Step S4: perform communication parallel pipeline data stream transmission on the asynchronous message communication queue to generate asynchronous parallel transmission information data; perform performance bottleneck analysis on the asynchronous parallel transmission information data to generate performance bottleneck analysis data; perform load regulation on the asynchronous parallel transmission information data through the performance bottleneck analysis data to generate multi-core concurrent load adjustment data.

[0029] The present invention can improve the parallelism of tasks by splitting and virtualizing tasks, so that multiple computing resources in the cluster can process different tasks at the same time, and speed up the overall processing speed. Through available resource analysis and ultra-fine-grained task division, the resources of each client in the cluster can be more fully utilized to avoid resource waste and bottlenecks. Meticulous task division and resource allocation help to improve the overall efficiency of the system, ensure that tasks can be completed in the shortest time, and better adapt to the heterogeneity of different clients in the cluster. Through the task type judgment rule engine and priority sorting, the priority of tasks can be effectively managed to ensure that urgent tasks are processed in time and improve the responsiveness of the system. Intelligent cache task scheduling can better utilize cluster resources, avoid resource waste, and improve the resource utilization and performance of the system. Multi-core concurrent task scheduling result data can guide the concurrent execution of tasks in a multi-core environment, improve the concurrent execution efficiency of the system, and accelerate the task processing speed. The use of high-order asynchronous message communication and task decoupling can improve the flexibility and performance of the system, making the system easier to expand and maintain. Through task decoupling, the dependency between the various modules of the system is reduced, and the maintainability and scalability of the system are improved. The use of non-blocking communication methods can improve the concurrency and responsiveness of the system, so that the system can handle a large number of task requests more effectively. Through asynchronous parallel transmission of information data generation, the system can utilize the ability of parallel processing to improve the efficiency and throughput of data transmission. Through performance bottleneck analysis and load regulation, the system can timely discover and solve performance bottleneck problems, thereby optimizing the performance of the system and improving the overall operating efficiency. The generated multi-core concurrent load adjustment data can help the system adjust the load in real time, and flexibly allocate resources according to the current operating status and performance requirements to ensure the stability and efficiency of the system under different workloads. Therefore, the present invention improves the concurrent processing capability and overall performance of the system by performing task splitting, intelligent scheduling, asynchronous communication, and performance analysis on cluster file client task data.

[0030] In the embodiment of the present invention, reference Figure 1The above is a schematic diagram of the steps of a method for implementing multi-core concurrent load on a cluster file client of the present invention. In this example, the method for implementing multi-core concurrent load on a cluster file client includes the following steps:

[0031] Step S1: Acquire cluster file client task data; perform task splitting on the cluster file client task data to generate subtask sorting data; perform available resource analysis on the cluster file client to generate client available resource data; perform virtualized ultra-fine-grained task division on the subtask sorting data according to the client available resource data to generate ultra-fine-grained virtual task list data;

[0032] In an embodiment of the present invention, task data is obtained from a cluster by using a corresponding cluster management tool (such as Apache Hadoop, Kubernetes, etc.) or a custom script. This may involve file system access and data extraction. According to the nature of the task, a script or program is written to split the large task into smaller subtasks and generate corresponding sorting data. This can be divided and sorted by specific attributes of the task. Using the API or monitoring tool provided by the cluster management tool, the available resource information of the cluster file client, including CPU, memory, storage, etc., is obtained. This information is organized into a data structure for subsequent use. Based on the client available resource data, an algorithm or rule is designed to further divide the subtasks into ultra-fine-grained virtual tasks. This may involve factors such as task priority and resource requirements. The information of the virtual task is organized into a data structure to form an ultra-fine-grained virtual task list. This list may include information such as task ID, priority, resource requirements, etc.

[0033] Step S2: construct a type judgment rule engine for the ultra-fine-grained virtual task list data to generate a task type judgment rule engine; prioritize the ultra-fine-grained virtual task list data through the task type judgment rule engine to generate urgent task priority ranking data; perform intelligent cache task scheduling on the urgent task priority ranking data to generate multi-core concurrent task scheduling result data;

[0034] In the embodiment of the present invention, by defining task type judgment rules, these rules can be judged based on the attributes, requirements, execution time, etc. of the task. For example, the type of task can be determined based on factors such as the urgency of the task, resource requirements, and execution time. Use a rule engine library (such as Drools, Jess, etc.) or a custom rule engine to convert the defined rules into code. The rule engine can automatically judge the type of the task based on the input task information and return the corresponding result. The ultra-fine-grained virtual task list data is input into the task type judgment rule engine to obtain the type of each task. According to the task type and other related factors (such as the urgency of the task, resource requirements, etc.), a sorting algorithm or rule is designed to prioritize the tasks. This can use various sorting algorithms, such as sorting based on priority queues, weighted sorting, etc. Design an intelligent cache task scheduling algorithm, which takes into account factors such as the type, priority, and available resources of the task, and allocates the task to the multi-core concurrent processing unit in the cluster in the best way. Optimization algorithms such as dynamic programming, greedy algorithms, and genetic algorithms can be used to achieve intelligent scheduling. These algorithms need to consider factors such as dependencies between tasks and resource contention. Based on the output of the intelligent cache task scheduling algorithm, the result data of multi-core concurrent task scheduling is generated. This data includes information such as which processing unit each task is assigned to and the execution order. The scheduling result data is saved in an appropriate data structure for subsequent execution or display.

[0035] Step S3: performing high-order asynchronous message communication on the multi-core concurrent task scheduling result data to generate asynchronous message communication data; performing task decoupling on the asynchronous message communication data to generate asynchronous message decoupling data; performing non-blocking communication on the asynchronous message decoupling data to generate an asynchronous message communication queue;

[0036] In an embodiment of the present invention, the multi-core concurrent task scheduling result data is converted into asynchronous message communication data. This can be achieved through mechanisms such as message queues or event buses. Select appropriate message transmission protocols and communication methods, such as AMQP (Advanced Message Queuing Protocol), MQTT (Message Queuing Telemetry Transport), Kafka, etc. These protocols and technologies support asynchronous message communication and can provide efficient message transmission mechanisms. Design a task decoupling mechanism to decouple tasks in asynchronous message communication data for subsequent processing and distribution. The publish-subscribe mode or message router can be used to achieve task decoupling. In this mode, the producer of the task publishes the message to the specified topic or route, and the consumer of the task subscribes to the topic or route of interest, thereby achieving task decoupling and distribution. The decoupled asynchronous message data is processed by non-blocking communication. This can be achieved through technologies such as asynchronous I / O and event loops. Using non-blocking communication methods can improve the throughput and response speed of the system and avoid thread blocking caused by waiting for I / O operations. The decoupled message data is stored in the asynchronous message communication queue for subsequent processing and scheduling. You can use message queue services (such as RabbitMQ, Apache Kafka, Amazon SQS, etc.) or custom message queue implementations to manage asynchronous message communication queues. These services provide reliable message storage and transmission mechanisms and support high concurrency and distributed deployment.

[0037] Step S4: perform communication parallel pipeline data stream transmission on the asynchronous message communication queue to generate asynchronous parallel transmission information data; perform performance bottleneck analysis on the asynchronous parallel transmission information data to generate performance bottleneck analysis data; perform load regulation on the asynchronous parallel transmission information data through the performance bottleneck analysis data to generate multi-core concurrent load adjustment data.

[0038] In an embodiment of the present invention, a communication parallel pipeline is designed and implemented to transfer information data between asynchronous message communication queues. This may include steps such as data format definition, serialization and deserialization. Consider using efficient serialization tools, such as Protocol Buffers, MessagePack, etc., to reduce the overhead of data transmission. Ensure that the design of the communication parallel pipeline can meet the concurrency requirements of the system and support multi-core concurrent tasks while maintaining communication efficiency. Monitor the performance indicators of asynchronous parallel transmission of information data, including but not limited to latency, throughput, system resource utilization, etc. Use performance analysis tools and monitoring systems, such as Prometheus, Grafana, etc., to collect and analyze performance data. Identify potential performance bottlenecks, which may involve communication pipelines, message queue services, data serialization, network bandwidth, etc. According to the results of the performance bottleneck analysis, load regulation is performed. This may include adjusting the number of concurrent communication pipelines, optimizing the configuration of message queues, improving data serialization performance, etc. In a multi-core concurrent environment, it is possible to consider dynamically adjusting the task distribution strategy to ensure that each core is fully utilized to avoid load imbalance. Use automated tools or algorithms to monitor system load and performance, and dynamically adjust according to real-time data to adapt to changes in system runtime. Record the results of load regulation and adjustment parameters to generate multi-core concurrent load adjustment data. This data can be used to further analyze the performance optimization and adjustment effects of the system, and can also be used as a reference for future system adjustments.

[0039] Preferably, step S1 comprises the following steps:

[0040] Step S11: Obtain cluster file client task data;

[0041] Step S12: performing task data flow relationship analysis on the cluster file client task data to obtain cluster file client relationship data; constructing a task topology structure based on the cluster file client relationship data to generate initial task topology structure data;

[0042] Step S13: splitting the initial task topology data into tasks to generate subtask sequence data; sorting the subtask sequence data to generate subtask sorting data; using the initial task topology data to analyze the available resources of the cluster file client to generate client available resource data;

[0043] Step S14: perform computational resource mapping on the subtask sorting data according to the client's available resource data to generate subtask resource mapping data; perform resource table allocation on the subtask resource mapping data to generate a task-resource mapping table; perform virtualized ultra-fine-grained task division on the subtask resource mapping data based on the task resource mapping table to generate ultra-fine-grained virtual task list data.

[0044] By acquiring task data, the system can accurately understand the current set of tasks to be processed and provide a data basis for subsequent steps. It is ensured that the system obtains complete task data from the cluster file client and provides an accurate data source for subsequent steps. By analyzing the task data flow relationship and constructing the task topology structure, the system can understand the dependencies between tasks, so as to better perform task scheduling and resource allocation. The initial task topology structure data is generated, which helps the system to more effectively manage the relationship between tasks and improve the efficiency of task scheduling. Through task splitting and sorting, the system can decompose large tasks into smaller subtasks and sort them according to resource conditions to optimize the task execution order. Resource analysis helps to understand the available resource conditions of the cluster file client. Subtask sequence data and subtask sorting data, as well as client available resource data, are generated, which provide a basis for subsequent resource mapping and task allocation. By calculating resource mapping and task allocation, the system can better utilize the computing resources of the cluster file client and improve the overall task execution efficiency. Ultra-fine-grained task division can manage tasks more finely and improve resource utilization. Subtask resource mapping data and task-resource mapping tables, as well as ultra-fine-grained virtual task list data, are generated to provide detailed guidance and planning for task execution, maximizing resource utilization and task execution efficiency.

[0045] In the embodiment of the present invention, a module or service is set in the system to obtain task data from the cluster file client. This may involve protocols and interfaces for communicating with the client to ensure the security and reliable transmission of data. A data flow relationship analysis algorithm is implemented, which can detect the dependencies and data flow between tasks. According to the data flow relationship obtained by analysis, the topological structure of the task is constructed, which can be implemented by using relevant algorithms in graph theory, such as depth-first search (DFS) or breadth-first search (BFS). A task splitting algorithm is implemented to split large tasks into smaller subtasks to ensure that each subtask can be executed in parallel. The subtasks are sorted to optimize the order of task execution, which can usually be sorted according to factors such as task priority and dependency. The client available resource analysis is performed to detect the computing resource status of the cluster file client, including CPU, memory, storage, etc. According to the client available resource data, the subtasks are mapped to appropriate computing resources to ensure that the tasks can be efficiently executed in the cluster. The resources are allocated to ensure that each task has sufficient resources to support its execution. An ultra-fine-grained task partitioning algorithm is implemented to further subdivide the tasks into smaller subtasks to improve the precision and efficiency of task execution.

[0046] Preferably, step S2 comprises the following steps:

[0047] Step S21: extracting concurrent task features from the ultra-fine-grained virtual task list data to generate concurrent task feature extraction data, wherein the concurrent task feature extraction data includes task type data, task priority data, and task dependency relationship data;

[0048] Step S22: construct a type judgment rule engine based on the concurrent task feature extraction data to generate a task type judgment rule engine; classify and prioritize the ultra-fine-grained virtual task list data through the task type judgment rule engine to generate cluster file task judgment data, wherein the cluster file task judgment data includes task-intensive judgment data and IO-intensive judgment data;

[0049] Step S23: Calculate the task urgency weight of the task-intensive judgment data and the IO-intensive judgment data according to the task urgency weight analysis formula to obtain the task urgency weight; prioritize the task-intensive judgment data and the IO-intensive judgment data using the task urgency weight to generate urgent task priority ranking data;

[0050] Step S24: formulate an intelligent cache strategy for the priority ranking data of urgent tasks, and generate a multi-core task intelligent cache strategy; schedule the priority ranking data of urgent tasks according to the multi-core task intelligent cache strategy, and generate multi-core concurrent task scheduling result data.

[0051] The present invention can extract important features such as the type, priority and dependency of tasks through concurrent task feature extraction. This helps the system to better understand and analyze the attributes of tasks, and provides a basis for subsequent task scheduling and resource allocation. The constructed task type judgment rule engine can classify and prioritize tasks according to their characteristics. Such an engine can allocate appropriate resources and scheduling strategies to tasks according to their different needs and characteristics, thereby improving the efficiency and performance of the system. The emergency weight of tasks is calculated by the task emergency weight analysis formula, and the tasks are prioritized. This can ensure that the system gives priority to tasks with higher urgency, and improves the system's response speed and processing efficiency for emergency tasks. The formulated multi-core task intelligent caching strategy can reasonably schedule tasks to be executed in a multi-core environment according to the urgency and characteristics of the tasks. Such a strategy can make full use of system resources, improve task concurrent processing capabilities, reduce system load, and thus improve the overall system performance and throughput.

[0052] As an example of the present invention, refer to Figure 2 As shown, in this example, step S2 includes:

[0053] Step S21: extracting concurrent task features from the ultra-fine-grained virtual task list data to generate concurrent task feature extraction data, wherein the concurrent task feature extraction data includes task type data, task priority data, and task dependency relationship data;

[0054] In an embodiment of the present invention, by ensuring that the preparation of ultra-fine-grained virtual task list data is completed, including information such as task description, type, priority and dependency. This information can come from a task management system, a database or other data source. Analyze the description of the task or other related information to identify the type of task. The task type can be determined based on the operation type, resource requirements, processing method, etc. of the task. Natural language processing technology, machine learning models or rule engines can be used to automatically identify the task type. Determine the priority of the task based on factors such as the importance, urgency, and deadline of the task. The priority of the task can be determined based on preset rules, business requirements or historical data, or a machine learning model can be used to predict the priority. Analyze the dependencies between tasks to determine the execution order or dependency between tasks. The dependencies between tasks can be determined based on information such as the input-output relationship and execution conditions of the task, or the dependencies between tasks can be analyzed by methods such as graph theory. Organize the extracted information such as task type, priority and dependency to generate concurrent task feature extraction data. This information can be stored in a data structure, such as a dictionary, a table or a database, for subsequent processing and analysis.

[0055] Step S22: construct a type judgment rule engine based on the concurrent task feature extraction data to generate a task type judgment rule engine; classify and prioritize the ultra-fine-grained virtual task list data through the task type judgment rule engine to generate cluster file task judgment data, wherein the cluster file task judgment data includes task-intensive judgment data and IO-intensive judgment data;

[0056] In the embodiment of the present invention, a task type judgment rule engine is constructed based on the extracted concurrent task feature data, including information such as task description, type, priority, etc. The rule engine can be a collection of a series of rules, each rule containing conditions and operations, which are used to make judgments and perform corresponding operations based on input data. Special rule engine software such as Drools, Jess, etc. can be used, or a rule engine module can be developed by oneself. According to the feature data of the task, a series of rules are defined to judge the type of the task. The rule can determine the task type based on features such as task description, input and output type, and resource requirements. For example, if the task description contains words such as "calculation" and "analysis", it is determined to be a computing task; if the task requires a large number of IO operations, it is determined to be an IO-intensive task. The defined rules are converted into code logic to realize the task type judgment function. In the rule engine, the priority of the rule can be defined to ensure that when multiple rules are matched, the corresponding operation is performed according to the priority level. Using the constructed task type judgment rule engine, the ultra-fine-grained virtual task list data is classified and the priority is judged. For each task, input its feature data into the rule engine, and determine the type and priority of the task through the judgment of the rule engine. According to the result of the task type judgment, generate cluster file task judgment data, including task-intensive judgment data and IO-intensive judgment data. Task-intensive judgment data may include a list of computationally intensive tasks, and IO-intensive judgment data may include a list of tasks that require a large number of IO operations. Store the generated cluster file task judgment data in a suitable data structure, such as a dictionary, database, etc., for subsequent task scheduling and management.

[0057] Step S23: Calculate the task urgency weight of the task-intensive judgment data and the IO-intensive judgment data according to the task urgency weight analysis formula to obtain the task urgency weight; prioritize the task-intensive judgment data and the IO-intensive judgment data using the task urgency weight to generate urgent task priority ranking data;

[0058] In an embodiment of the present invention, an analysis formula for task urgency weight is defined. This formula can calculate the urgency of a task based on various attributes of the task, such as deadline, task importance, resource requirements and other factors. The specific formula can be determined according to actual conditions and needs. According to the determined task urgency weight analysis formula, an urgency weight calculation is performed for each task in the task-intensive judgment data and the IO-intensive judgment data. It is necessary to input the relevant information of the task into the urgency weight calculation formula to calculate the urgency weight value of each task. The calculated task urgency weight value is used as a basis to prioritize the tasks in the task-intensive judgment data and the IO-intensive judgment data. Generally, the higher the task urgency weight value, the higher its priority and should be processed earlier. Therefore, the tasks can be sorted from high to low according to the size of the task urgency weight value. The sorted task list or data structure is saved as the priority sorting data of the urgent task. These data can be used by the task scheduling system or other related systems to ensure that urgent tasks are given priority.

[0059] Step S24: formulate an intelligent cache strategy for the priority ranking data of urgent tasks, and generate a multi-core task intelligent cache strategy; schedule the priority ranking data of urgent tasks according to the multi-core task intelligent cache strategy, and generate multi-core concurrent task scheduling result data.

[0060] In an embodiment of the present invention, an intelligent cache strategy is designed to maximize the task execution efficiency on a multi-core processor. This may involve the design of a task scheduling algorithm and a cache management strategy. The cache area of ​​the multi-core processor is divided into multiple parts, each of which is used to store task data of different priorities. This ensures that the data of high-priority tasks are preferentially stored in the cache, thereby improving their access efficiency. Design appropriate data pre-fetching and replacement strategies to reduce cache miss rates and minimize cache access delays. For example, a pre-fetching algorithm based on task access patterns may be used, or a replacement strategy may be dynamically adjusted according to task priorities. When formulating intelligent cache strategies, the concurrency of tasks needs to be considered. Even for urgent tasks, there may be data dependencies or resource competition, and it is necessary to design appropriate concurrency control strategies to ensure that tasks can be effectively executed in parallel. According to the formulated multi-core task intelligent cache strategy, urgent task priority ranking data is scheduled. This may involve dynamic allocation of tasks, priority adjustment, and management of concurrent execution. Design a task scheduling algorithm to reasonably allocate tasks to multi-core processors based on factors such as task urgency, resource requirements, data dependencies, etc., and take into account cache usage and task concurrency requirements. Generate multi-core concurrent task scheduling result data based on the execution results of the task scheduling algorithm. These data will guide the actual task execution process and ensure that tasks can be executed efficiently according to priority sorting data and intelligent caching strategies.

[0061] Preferably, the task urgency weight analysis formula in step S23 is as follows:

[0062]

[0063] Where U represents the task urgency weight, P represents the task priority coefficient, R represents the task resource requirement coefficient, E represents the expected efficiency of the task, T represents the task deadline, C represents the computational amount of the task, D represents the data volume of the task, M represents the task-intensive judgment data, I represents the IO-intensive judgment data, and x represents the input variable.

[0064] The present invention analyzes and integrates a task emergency weight analysis formula, in which It reflects the impact of the task priority P and resource requirement R on the task urgency. If the task priority P is higher, its urgency will increase. At the same time, if the task's resource requirement R is greater, the task urgency will also increase. This relationship reflects the contribution of the task's importance and the urgency of resource requirements to the task urgency. The calculation amount C and data amount D of the task are integrated. Here, it is assumed that the calculation amount C is an indicator representing the computing resource requirements of the task, and the data amount D represents the data processing amount of the task. By integrating the square root of the calculation amount and the exponential function of the data amount, a complexity index of the task can be obtained. This index can reflect the complexity of the calculation and data processing of the task, thereby affecting the urgency of the task. The more complex the calculation and data processing of the task, the greater its urgency. Consideration is given to whether the task is both task-intensive M and IO-intensive I. If the task is both task-intensive and IO-intensive, its urgency will increase. This is because the task has higher demands on both computing resources and IO resources, and requires more attention and processing. When using the conventional task urgency weight analysis formula in the field, the task urgency weight can be obtained. By applying the task urgency weight analysis formula provided by the present invention, the task urgency weight can be calculated more accurately. The purpose of designing the formula is to comprehensively consider multiple factors such as task priority, resource requirements, expected efficiency, deadline, computational load, data volume, task intensiveness and IO intensiveness, so as to calculate the urgency weight of the task.

[0065] Preferably, step S24 includes the following steps:

[0066] Step S241: performing task cache access feature analysis on the emergency task priority ranking data to generate task access feature data, wherein the task access feature data includes file size data, access frequency data, and most recent access time data; importing the file size data, access frequency data, and most recent access time data into a preset deep reinforcement learning model to perform learning environment modeling, and generating a reinforcement learning environment model;

[0067] Step S242: Divide the task access feature data into data sets to generate a model training set and a model test set; perform model training on the reinforcement learning environment model using the model training set to generate a concurrent throughput training model; perform model testing on the concurrent throughput training model using the model test set to generate a concurrent throughput prediction model;

[0068] Step S243: importing the task access feature data into the concurrent throughput prediction to generate concurrent throughput prediction data; comparing the concurrent throughput prediction data with the preset throughput threshold range, when the concurrent throughput prediction data is greater than the preset throughput threshold range, generating a first task cache strategy; when the concurrent throughput prediction data is within the preset throughput threshold range, generating a second task cache strategy; when the concurrent throughput prediction data is less than the preset throughput threshold range, generating a third task cache strategy;

[0069] Step S244: when the first task cache strategy is confirmed, the cache capacity of the urgent task priority ranking data is increased to generate cache capacity increase data; the urgent task priority ranking data is cached with high frequency priority using the cache capacity increase data to generate first task cache increase data;

[0070] Step S245: When the second task cache strategy is confirmed, the cache cycle analysis is performed on the emergency task priority sorting data to generate the emergency task priority cache cycle data; the emergency task priority cache cycle data is periodically cache cleaned using the LRU algorithm to generate the second task cache cleanup data;

[0071] Step S246: when the third task cache strategy is confirmed, priority task access restriction is performed on the urgent task priority sorting data to generate urgent task priority access restriction data; cache pre-fetch is performed on the urgent task priority sorting data according to the urgent task priority access restriction data to generate third task cache restriction data;

[0072] Step S247: Integrate the cache strategy of the first task cache increase data, the second task cache clearing data and the third task cache restriction data to generate a multi-core task intelligent cache strategy; schedule the emergency task priority sorting data according to the multi-core task intelligent cache strategy to generate multi-core concurrent task scheduling result data.

[0073] The present invention analyzes the access characteristics of task caches for emergency task priority sorting data, and combines the deep reinforcement learning model to model the learning environment, so as to more accurately predict the concurrent throughput, thereby optimizing task scheduling. Different task cache strategies are generated according to the concurrent throughput prediction data, and different strategies are adopted for different situations, such as increasing the cache capacity, regular cache cleaning, and priority task access restrictions, which are helpful to improve the task execution efficiency and the overall performance of the system. Through the analysis and prediction of the task access characteristic data, the system resources can be more effectively utilized, the waste of resources can be reduced, and the utilization rate of system resources can be improved. The optimized task scheduling and cache strategy can reduce the system load, reduce resource competition and waiting time, and improve the system response speed and stability. Comprehensively considering the integration of task characteristics, concurrent throughput prediction and cache strategy, the overall performance of the multi-core system can be maximized, and more efficient task execution and resource management can be achieved. Due to the prediction and adjustment mechanism based on the learning model, the system has a certain degree of adaptability, and can be dynamically adjusted according to the real-time workload and environmental changes to maintain efficient operation in different scenarios.

[0074] As an example of the present invention, refer to Figure 3 As shown, in this example, step S24 includes:

[0075] Step S241: performing task cache access feature analysis on the emergency task priority ranking data to generate task access feature data, wherein the task access feature data includes file size data, access frequency data, and most recent access time data; importing the file size data, access frequency data, and most recent access time data into a preset deep reinforcement learning model to perform learning environment modeling, and generating a reinforcement learning environment model;

[0076] In an embodiment of the present invention, by collecting urgent task priority sorting data, including information such as task identification, file size, access frequency, and last access time. The collected data is analyzed to extract features related to task cache access. Possible features include file size, access frequency, last access time, etc. Based on the urgency of the task, the tasks are prioritized for subsequent modeling and decision-making. An appropriate deep reinforcement learning model, which may be a neural network-based model, is selected to learn the environment model of the task. The extracted task access feature data, such as file size, access frequency, and last access time, are preprocessed to adapt to the input format of the deep learning model. The preprocessed data is input into the deep reinforcement learning model for training to learn the model of the task environment. The goal of the training is to enable the model to accurately predict the impact of access features of different tasks on system performance. The trained model is verified to ensure that it has good generalization ability on unseen data. New task data, including features such as file size, access frequency, and last access time, are imported into the trained deep reinforcement learning model. Through the forward propagation of the model, a reinforcement learning environment model is generated, which can describe the impact of different task features on cache access. The output of the model can be a value indicating the priority of the task or the importance of affecting cache access, or a set of action incentives for subsequent decision making.

[0077] Step S242: Divide the task access feature data into data sets to generate a model training set and a model test set; perform model training on the reinforcement learning environment model using the model training set to generate a concurrent throughput training model; perform model testing on the concurrent throughput training model using the model test set to generate a concurrent throughput prediction model;

[0078] In an embodiment of the present invention, the task access feature data is divided into a model training set and a model test set according to a certain ratio. Generally, 70% of the data can be used as a training set and 30% of the data can be used as a test set. Ensure that the data distribution of the training set and the test set represents the distribution of the overall data as much as possible. Use the data of the model training set to train with the selected deep reinforcement learning model. During the training process, the model parameters are optimized by minimizing the loss function so that it can accurately learn the characteristics and dynamics of the task environment. The goal of the training is to enable the model to effectively predict the impact of task characteristics on concurrent throughput. The trained concurrent throughput training model is tested using the data of the model test set. The test set data is input into the model to evaluate the model's predictive ability for concurrent throughput. Various evaluation indicators, such as mean squared error (MSE), mean absolute error (MAE), etc., can be used to evaluate the performance of the model. According to the results of the model test, the performance and accuracy of the model are evaluated. If the model performs well and meets expectations, it can be used to predict concurrent throughput. If the performance of the model is not satisfactory, you can consider adjusting the model's hyperparameters, improving the feature extraction method, or trying other deep learning model structures to optimize the model. The tested and verified model can be used in the actual concurrent throughput prediction task. By inputting the new task feature data into the trained model, the corresponding concurrent throughput prediction result can be obtained.

[0079] Step S243: importing the task access feature data into the concurrent throughput prediction to generate concurrent throughput prediction data; comparing the concurrent throughput prediction data with the preset throughput threshold range, when the concurrent throughput prediction data is greater than the preset throughput threshold range, generating a first task cache strategy; when the concurrent throughput prediction data is within the preset throughput threshold range, generating a second task cache strategy; when the concurrent throughput prediction data is less than the preset throughput threshold range, generating a third task cache strategy;

[0080] In an embodiment of the present invention, task access feature data is collected through a monitoring system or other data collection methods. These data may include the type of task, access frequency, access mode, etc. The concurrent throughput of the task is predicted using machine learning, statistical analysis or other prediction models. This may involve analysis and pattern recognition of historical data, as well as evaluation of system load and resource utilization. The predicted concurrent throughput is compared with a preset throughput threshold range. These thresholds may be based on factors such as system performance indicators, resource constraints or service level agreements (SLAs). If the predicted concurrent throughput is greater than the preset threshold range, a first task cache strategy is generated. This may include increasing the cache size, optimizing the cache replacement strategy, etc. If the predicted concurrent throughput is within the preset threshold range, a second task cache strategy is generated. This may mean that the current cache strategy is still valid and no change is required. If the predicted concurrent throughput is less than the preset threshold range, a third task cache strategy is generated. This may involve reducing the cache size, adjusting the cache replacement strategy, etc. to avoid resource waste.

[0081] Step S244: when the first task cache strategy is confirmed, the cache capacity of the urgent task priority ranking data is increased to generate cache capacity increase data; the urgent task priority ranking data is cached with high frequency priority using the cache capacity increase data to generate first task cache increase data;

[0082] In an embodiment of the present invention, by collecting urgent task priority sorting data and evaluating the current cache capacity. A cache capacity increase scheme is designed, which can be to increase the total capacity of the cache, or to adjust the cache quota of different tasks. According to the designed scheme, cache capacity increase data is generated, including information such as the increased capacity size and the increased task cache quota. The urgent task priority sorting data is merged with the cache capacity increase data to facilitate the subsequent generation of a high-frequency cache priority strategy. A high-frequency cache priority strategy based on cache capacity increase is designed, which can adjust the priority of the task in the cache according to the cache requirements and urgency of the task. According to the designed high-frequency cache priority strategy, the urgent task priority sorting data is processed to generate the first task cache increase data. The data includes information such as the task identifier selected as the first task cache and the corresponding cache increase amount.

[0083] Step S245: When the second task cache strategy is confirmed, the cache cycle analysis is performed on the emergency task priority sorting data to generate the emergency task priority cache cycle data; the emergency task priority cache cycle data is periodically cache cleaned using the LRU algorithm to generate the second task cache cleanup data;

[0084] In the embodiment of the present invention, the second task cache strategy is explicitly adopted and its basic principles are understood. This may include determining which tasks are considered urgent tasks and how to prioritize them. Based on the definition of urgent tasks and business needs, the priority sorting method of tasks is determined. This can be sorting according to the importance, urgency or other specific criteria of the tasks. The cache cycle of each task is analyzed, that is, how long each task needs to be kept in the cache is determined. This may be based on factors such as the nature of the task and the access frequency. The priority of the task and the corresponding cache cycle data are sorted into a data set for use by the subsequent cache cleaning algorithm. The priority of the task and the corresponding cache cycle data are sorted into a data set for use by the subsequent cache cleaning algorithm. The LRU (Least Recently Used) algorithm is selected as the basic algorithm for cache cleaning. The LRU algorithm eliminates the least recently used cache items and is suitable for situations with limited space in the cache. The LRU algorithm is implemented in the cache, and the least frequently accessed tasks in the cache are regularly cleaned according to the priority and cache cycle data of the tasks. This ensures that urgent tasks are always retained in the cache and sufficient space is provided for them. The tasks involved in the cache cleaning process and the time points of the cleaning are recorded to generate the second task cache cleaning data.

[0085] Step S246: when the third task cache strategy is confirmed, priority task access restriction is performed on the urgent task priority sorting data to generate urgent task priority access restriction data; cache pre-fetch is performed on the urgent task priority sorting data according to the urgent task priority access restriction data to generate third task cache restriction data;

[0086] In an embodiment of the present invention, the third task cache strategy is adopted by confirming and understanding its basic principles. Determining the priority access restriction of the task is a key step of the strategy. According to business requirements and system performance considerations, the priority access restriction rules of the urgent task are defined. This may include limiting the access frequency, access time period, etc. of certain tasks. The priority access restriction rules are applied to the priority sorting data of the task to generate a data set containing access restriction information. This data set will guide the subsequent cache pre-fetching steps. Select an appropriate cache pre-fetching algorithm to ensure that the data of the urgent task is loaded into the cache in advance under the premise of complying with the access restrictions. This can be optimized based on the access frequency, the predictive nature of the task, etc. According to the priority access restriction data of the urgent task, the data of the urgent task is loaded into the cache in advance using the selected cache pre-fetching algorithm. This helps to respond more quickly when the actual request arrives. Record the tasks involved in the cache pre-fetching process and the time points of the pre-fetching to generate the third task cache restriction data.

[0087] Step S247: Integrate the cache strategy of the first task cache increase data, the second task cache clearing data and the third task cache restriction data to generate a multi-core task intelligent cache strategy; schedule the emergency task priority sorting data according to the multi-core task intelligent cache strategy to generate multi-core concurrent task scheduling result data.

[0088] In an embodiment of the present invention, the first task cache increase data, the second task cache clearing data and the third task cache limit data are collected and integrated into a unified data structure. Based on the integrated data, a multi-core task intelligent caching strategy is designed. This strategy may involve the priority relationship between different tasks, the update frequency of cache data, the clearing strategy, etc. The urgent task priority sorting data is analyzed to understand the priority and characteristics of different tasks. Based on the multi-core task intelligent caching strategy, a task scheduling strategy is formulated. This may include determining the execution order of tasks, the distribution method on the multi-core processor, etc. According to the formulated scheduling strategy, the urgent task priority sorting data is task scheduled to generate multi-core concurrent task scheduling result data.

[0089] Preferably, step S3 comprises the following steps:

[0090] Step S31: performing high-order asynchronous message passing framework setting on the multi-core concurrent task scheduling result data to generate a high-order asynchronous message passing framework, wherein the high-order asynchronous message passing framework includes a message producer node and a message consumer node;

[0091] Step S32: transmitting the multi-core concurrent task scheduling result data to the message producer node for task message transmission to generate multi-core concurrent task transmission data; performing asynchronous message communication on the multi-core concurrent task transmission data through API calls to generate asynchronous message communication data;

[0092] Step S33: using the message consumer node to perform asynchronous message processing on the asynchronous message communication data to generate asynchronous message processing data;

[0093] Step S34: Decouple the asynchronous message processing data and the multi-core concurrent task scheduling result data to generate asynchronous message decoupled data; perform non-blocking communication on the asynchronous message decoupled data through the message producer node and the message consumer node to generate an asynchronous message communication queue.

[0094] The present invention can make full use of the computing power of multi-core processors and improve the overall performance and throughput of the system through concurrent task scheduling and asynchronous message processing. The use of asynchronous message transmission framework and message queue makes task scheduling and processing more flexible and configurable, and can cope with changes in demand in different scenarios. Due to the use of distributed message transmission and processing, the system has good scalability and can be horizontally expanded as the task scale and load increase. Abstracting task decoupling and data exchange into a message transmission method can reduce the coupling and complexity of the system and improve the maintainability and scalability of the system.

[0095] As an example of the present invention, refer to Figure 4 As shown, in this example, step S3 includes:

[0096] Step S31: performing high-order asynchronous message passing framework setting on the multi-core concurrent task scheduling result data to generate a high-order asynchronous message passing framework, wherein the high-order asynchronous message passing framework includes a message producer node and a message consumer node;

[0097] In the embodiment of the present invention, by evaluating different high-order asynchronous messaging frameworks, such as RabbitMQ, Apache Kafka, ActiveMQ, etc., the framework that best suits the project requirements is selected. Consider factors such as the performance, scalability, reliability, and community support of the framework. Determine which components or modules will act as message producer nodes, usually components related to task scheduling. For the task scheduling result data, write code or configuration to convert the data into messages. Determine the format and structure of the message, and implement corresponding data serialization and formatting in the producer node. Deploy the selected messaging framework to ensure the availability and reliability of the framework. Configure the messaging framework to adapt to the project requirements, including setting parameters such as message queues, topics, and persistence. Ensure the security of the messaging framework and take necessary authentication and authorization measures to prevent unauthorized access. Determine which components or modules will act as message consumer nodes, usually components related to task execution or result processing. Write code or configuration to connect to the messaging framework and subscribe to the corresponding message queue or topic. Implement message processing logic in the consumer node, including operations such as message deserialization, data processing, and storage.

[0098] Step S32: transmitting the multi-core concurrent task scheduling result data to the message producer node for task message transmission to generate multi-core concurrent task transmission data; performing asynchronous message communication on the multi-core concurrent task transmission data through API calls to generate asynchronous message communication data;

[0099] In an embodiment of the present invention, the location and role of the message producer node are determined. These nodes are usually located at the location where the task scheduling result data is generated or in a component that is closely integrated with the task scheduling system. The environment of the message producer node is configured, including the software and dependencies required for installing and configuring the messaging framework (such as RabbitMQ, Kafka, etc.). In a multi-core concurrent task scheduling system, code is written or rules are set to ensure that the task scheduling result data is transmitted to the message producer node. This can be achieved by directly triggering a message sending operation in the task scheduling system, or by periodically polling the task scheduling results and sending them to a message queue or topic. According to the format and content of the task scheduling result data, it is converted into a format suitable for messaging. This may involve operations such as serialization, structural adjustment, or additional metadata of the data. The converted data is published to the messaging framework for subsequent asynchronous message communication. Determine the API endpoints and functions that require asynchronous message communication. In the relevant API endpoints, code is written to trigger asynchronous message communication. This can be by calling the API provided by the messaging framework to publish the message to the corresponding message queue or topic. In the API call, the data to be transmitted is converted into a message format. Call the API provided by the messaging framework to publish messages to a message queue or topic to implement asynchronous message communication.

[0100] Step S33: using the message consumer node to perform asynchronous message processing on the asynchronous message communication data to generate asynchronous message processing data;

[0101] In the embodiment of the present invention, by selecting a suitable component or module in the system as a message consumer node, it is responsible for receiving asynchronous message communication data. Write code or configuration to connect to the messaging framework and subscribe to the corresponding message queue or topic. Implement asynchronous message processing logic in the message consumer node. This may include operations such as task scheduling, data processing, and result generation, depending on the needs of the project. Ensure that the message processing logic is non-blocking to give full play to the advantages of asynchronous message processing. Implement appropriate exception handling mechanisms to deal with possible errors and abnormal situations. This includes network failures, message format errors, etc. Consider introducing a retry mechanism to ensure that messages can be reprocessed in the event of a failure to improve the stability of the system. Parse and convert asynchronous message communication data in the message consumer node. Ensure that the data in the message can be correctly extracted for subsequent processing. Deserialize the data according to the message format and convert it into an internal data structure. Generate asynchronous message processing data based on the content and processing logic of the message. This may include information such as processing results and status updates. Ensure that the format and structure of the asynchronous message processing data meet the needs of the system.

[0102] Step S34: Decouple the asynchronous message processing data and the multi-core concurrent task scheduling result data to generate asynchronous message decoupled data; perform non-blocking communication on the asynchronous message decoupled data through the message producer node and the message consumer node to generate an asynchronous message communication queue.

[0103] In an embodiment of the present invention, the asynchronous message processing data and multi-core concurrent task scheduling result data that need to be decoupled are determined. These data are analyzed to determine the association relationship between each other and the decoupling method. This may involve splitting, reorganizing or converting the original data. Write code or configuration to implement the process of asynchronous message decoupling. This may include operations such as data segmentation, data filtering, and data reorganization. Ensure that the generated asynchronous message decoupling data can meet the needs of subsequent communication and maintain the integrity and consistency of the data. Select a suitable component or module as a message producer node, which is responsible for sending the asynchronous message decoupling data to the message communication queue. Write code or configuration to connect the messaging framework and publish the decoupled data to the corresponding message queue or topic. Similarly, select a suitable component or module as a message consumer node, which is responsible for receiving and processing asynchronous message decoupling data. Write code or configuration to connect the messaging framework and subscribe to the corresponding message queue or topic to receive the decoupled data. In the communication process between the message producer node and the message consumer node, ensure that data is transmitted in a non-blocking manner. This may involve the use of asynchronous IO, event-driven and other technologies to ensure that the communication process does not block other operations of the system. The asynchronous message decoupled data is sent to the message communication queue through the message producer node. The message consumer node receives the decoupled data from the queue and performs subsequent processing. Ensure that the configuration and performance of the message communication queue can meet the needs of the system, including the reliability, sequence and latency of the message.

[0104] Preferably, step S33 includes the following steps:

[0105] Step S331: using the message consumer node to receive the asynchronous message communication data to obtain the asynchronous message communication received data; performing data content analysis on the asynchronous message communication received data to generate message content analysis data;

[0106] Step S332: performing data cleaning on the message content parsing data to generate message content cleaning data; performing data format conversion on the message content cleaning data to generate message content format conversion data; performing data standardization on the message content format conversion data by using a maximum-minimum standardization method to generate standard message content data;

[0107] Step S333: updating the system status of the cluster file system based on the standard message content data to generate message content status update data; persistently storing the standard message content data according to the message content status update data to generate message content storage data;

[0108] Step S334: Perform asynchronous event triggering on the message content storage data to generate a message content triggering event; perform message processing notification on the cluster file system according to the message content triggering event, thereby generating asynchronous message processing data.

[0109] The present invention receives and parses asynchronous message communication data through message consumer nodes, thereby realizing effective acquisition and preliminary processing of asynchronous messages. It provides the basis of original data and provides parsed message content for subsequent steps. It cleans, converts the format and standardizes the parsed message content to ensure data consistency and availability. It provides standardized data for subsequent processing and simplifies the complexity in subsequent processing steps. It updates the status of the cluster file system according to the standard message content data and stores the data persistently. It realizes real-time update of the system status and long-term preservation of data, ensuring the accuracy and traceability of the system status. It generates a message content triggering event through asynchronous event triggering, and sends a message processing notification to the cluster file system. It starts the asynchronous processing flow to ensure that the system can perform corresponding processing when receiving a specific event, thereby realizing the automation and responsiveness of the system.

[0110] In an embodiment of the present invention, a message queue or a message middleware (such as RabbitMQ, Kafka, etc.) is used as a message consumer node to subscribe to asynchronous message communication data. The message consumer node is responsible for receiving asynchronous message communication data, parsing the data content, and generating message content parsing data. The client library of the message queue (such as pika, kafka-python, etc.) is used to communicate with the message queue, and common data parsing tools or custom parsing logic can be used to parse the data. Data cleaning, format conversion and standardization are performed on the message content parsing data. Data cleaning may include removing invalid data, filling missing values, etc.; format conversion may convert data into a specific format or data structure; standardization may use a maximum-minimum standardization method, etc. to scale the data to a specified range. The cluster file system is updated with the system status based on the standard message content data, and the data is persistently stored. The system status update can be achieved by calling the corresponding API or executing a specific system command; data persistence can be stored using a database or a file system. Use a suitable database (such as MySQL, MongoDB, etc.) for data persistence storage, and select a suitable storage solution according to the requirements. Asynchronous event triggering is performed on the message content storage data, and a message processing notification is sent to the cluster file system. Asynchronous event triggering can be achieved through message queues, and message processing notifications can be sent through the system's internal communication mechanism (such as API calls, WebSocket, etc.). Use message queues to implement asynchronous event triggering, such as pushing messages to specific message topics; use appropriate communication protocols (such as HTTP, WebSocket, etc.) to send processing notifications to the cluster file system.

[0111] Preferably, step S4 comprises the following steps:

[0112] Step S41: constructing a communication parallel pipeline for the asynchronous message communication queue to generate a communication parallel pipeline; using the communication parallel pipeline to transmit data streams to the asynchronous message communication queue to generate asynchronous parallel transmission information data;

[0113] Step S42: performing system performance monitoring on the asynchronous parallel transmission information data to obtain system performance monitoring data; adjusting the monitoring time period of the asynchronous parallel transmission information data according to the system performance monitoring data to generate monitoring time period data; collecting historical data of the asynchronous parallel transmission information data according to the monitoring time period data to obtain historical transmission information data;

[0114] Step S43: Perform performance bottleneck analysis on the historical information transmission data to generate performance bottleneck analysis data; perform load regulation on the asynchronous parallel information transmission data through the performance bottleneck analysis data to generate multi-core concurrent load adjustment data.

[0115] The present invention can improve the concurrent processing capability and throughput of the system by constructing communication parallel pipelines and using them to transmit data streams of asynchronous message communication queues. This helps to reduce the delay of message transmission and increase the system's processing capability for concurrent requests, thereby improving the performance and response speed of the system. System performance monitoring of asynchronous parallel transmission information data and adjustment based on the monitoring results can help the system monitor and evaluate its performance status in real time. By monitoring time cycle data and historical transmission information data, the system can better understand its operating status and trends, thereby timely discovering and solving potential performance problems and ensuring the stability and reliability of the system. Performance bottleneck analysis of historical transmission information data and load regulation of asynchronous parallel transmission information data based on the analysis results can help the system optimize resource utilization and adjust task allocation, thereby maximizing the performance and efficiency of the system. Through multi-core concurrent load adjustment data, the system can better utilize existing hardware resources, effectively solve performance bottleneck problems, and improve the overall performance of the system.

[0116] In the embodiment of the present invention, by selecting a suitable communication parallel pipeline technology, such as a message queue system or other asynchronous communication tools. According to the system requirements and architecture design, a communication parallel pipeline is established to ensure that it supports asynchronous message transmission. The communication parallel pipeline is used to implement the data flow transmission mechanism of the asynchronous message communication queue. Suitable parallel processing technology is adopted to ensure that messages can be efficiently and concurrently transmitted in the pipeline. An error handling mechanism for data flow transmission is implemented to ensure system stability. An integrated performance monitoring tool is used to monitor key performance indicators of asynchronous parallel transmission information data, such as response time, throughput, etc. The system performance monitoring data is recorded in real time and a performance monitoring report is established. According to the system performance monitoring data, the monitoring time period is dynamically adjusted to ensure that the system can flexibly adapt to the needs of different workloads. An adaptive algorithm or rule engine is used to realize automatic adjustment of the monitoring time period. According to the monitoring time period data, historical data of asynchronous parallel transmission information data is regularly collected. The historical data is stored in a queryable data storage system for subsequent performance analysis and optimization. The historical transmission information data is used to perform performance bottleneck analysis to identify system bottlenecks and performance bottlenecks. The performance analysis tool or algorithm is used to find out the key factors that cause performance degradation. According to the performance bottleneck analysis data, implement load regulation strategies, and adjust the processing method of asynchronous parallel transmission information data. Use multi-core concurrent load adjustment data to optimize system resource utilization and improve the system's concurrent processing capabilities. Implement a real-time load adjustment mechanism to ensure that the system can respond to sudden load changes in a timely manner. Combined with monitoring time period data, dynamically adjust the system configuration to achieve optimal performance and resource utilization.

[0117] Preferably, step S41 includes the following steps:

[0118] Step S411: analyzing the communication processing process of the asynchronous message communication queue to generate asynchronous message communication processing process data; dividing the pipeline process according to the asynchronous message communication processing process data to generate a communication parallel pipeline;

[0119] Step S412: allocating the asynchronous message communication processing process data to parallel processing units based on the communication parallel pipeline, and generating an asynchronous message parallel task allocation scheme;

[0120] Step S413: Implement the pipeline stage division diagram with parallel processing units according to the asynchronous message parallel task allocation scheme to generate parallel processing units; perform logical processing on the asynchronous message communication queue through the parallel processing units and based on the asynchronous message parallel task allocation scheme to generate asynchronous message logical data;

[0121] Step S414: Perform data flow transmission on the asynchronous message logic data to generate an asynchronous message logic processing data flow; merge the processing results of the asynchronous message logic processing data flow to generate asynchronous parallel transmission information data.

[0122] The present invention can improve the processing efficiency and performance of the system by analyzing and pipelining the processing process of the asynchronous message communication queue. The reasonable allocation of parallel processing units and the formulation of task allocation schemes are helpful to make full use of system resources and speed up the processing speed. Through the implementation of parallel processing units and the design of asynchronous message parallel task allocation schemes, the parallel processing of asynchronous message communication queues can be realized, and the concurrent processing capability of the system can be improved. Through the pipelined data processing flow, the dependency between processing tasks can be reduced and the parallelism of the system can be improved. The pipelined and parallelized design of the asynchronous message communication processing process makes the system more scalable. Through flexible parallel processing unit allocation and task allocation schemes, the system can adapt to the needs of different scales and loads. Through the data flow transmission of asynchronous message logical data and the merging of processing results, efficient processing and integration of data can be achieved. This helps to ensure that the data of the asynchronous message communication queue can be transmitted and processed as expected and obtain correct processing results. The design of the pipelined processing process and the parallel task allocation scheme makes the structure of the system clearer and easy to maintain and debug. The analysis and data generation of the asynchronous message communication processing process provide an effective basis for system performance tuning and troubleshooting.

[0123] In the embodiment of the present invention, the working mode of the asynchronous message communication queue is analyzed, including the generation, transmission and processing process of the message. According to the analysis result, the asynchronous message communication processing process data is generated, including the time, sequence, correlation and other information of the message generation, transmission and processing. Based on the asynchronous message communication processing process data, the pipeline process is divided, and the communication process is divided into different stages or steps to form a communication parallel pipeline. The generated communication parallel pipeline is analyzed to determine the part that can be processed in parallel. According to the structure and characteristics of the communication parallel pipeline, an appropriate number of parallel processing units are determined and allocated to achieve parallel processing. According to the allocation of the parallel processing units, an allocation scheme for the asynchronous message parallel tasks is formulated to ensure that each parallel processing unit is responsible for the appropriate task. According to the number of allocated parallel processing units and the task allocation scheme, the corresponding parallel processing unit is implemented, and multi-threading, distributed processing and other methods can be adopted. Using the implemented parallel processing unit, according to the asynchronous message parallel task allocation scheme, the asynchronous message communication queue is logically processed to generate asynchronous message logic data. The generated asynchronous message logic data is transmitted according to the predetermined data flow transmission mode to ensure the correct transmission and processing of the data. The asynchronous message logic processing data streams obtained by processing each parallel processing unit are merged to generate final asynchronous parallel transmission information data, which may include processing results, statistical information and the like.

[0124] In this specification, a cluster file system is provided for executing the above-mentioned cluster file client multi-core concurrent load implementation method, and the cluster file system includes:

[0125] The task division module is used to obtain the cluster file client task data; perform task division on the cluster file client task data to generate subtask sorting data; perform available resource analysis on the cluster file client to generate client available resource data; perform virtualized ultra-fine-grained task division on the subtask sorting data according to the client available resource data to generate ultra-fine-grained virtual task list data;

[0126] The judgment engine module is used to construct a type judgment rule engine for the ultra-fine-grained virtual task list data and generate a task type judgment rule engine; prioritize the ultra-fine-grained virtual task list data through the task type judgment rule engine and generate urgent task priority ranking data; perform intelligent cache task scheduling on the urgent task priority ranking data and generate multi-core concurrent task scheduling result data;

[0127] The asynchronous communication module is used to perform high-order asynchronous message communication on the multi-core concurrent task scheduling result data to generate asynchronous message communication data; perform task decoupling on the asynchronous message communication data to generate asynchronous message decoupling data; perform non-blocking communication on the asynchronous message decoupling data to generate an asynchronous message communication queue;

[0128] The load regulation module is used to transmit the communication parallel pipeline data stream of the asynchronous message communication queue to generate asynchronous parallel transmission information data; perform performance bottleneck analysis on the asynchronous parallel transmission information data to generate performance bottleneck analysis data; perform load regulation on the asynchronous parallel transmission information data through the performance bottleneck analysis data to generate multi-core concurrent load adjustment data.

[0129] The beneficial effect of the present invention is that subsequent processing is prepared by obtaining task data of a file client from a cluster. Task data is split into smaller subtasks and they are sorted. This helps to better manage tasks and improve execution efficiency. Available resources of a cluster file client are analyzed, including computing resources, storage resources, etc. This helps to reasonably allocate and schedule subsequent tasks. According to the available resource data of the client, the subtasks are further divided to generate an ultra-fine-grained virtual task list. In this way, the execution of tasks and resource utilization can be more finely controlled. Based on the ultra-fine-grained virtual task list data, a rule engine is built to determine the task type. In this way, more flexible processing can be performed according to the characteristics and requirements of the task. Tasks are prioritized according to the urgency of the task and other factors, and intelligent cache task scheduling is performed. This helps to ensure that urgent tasks can be processed in a timely manner and improve the response speed and efficiency of the system. High-order asynchronous message communication is performed on the multi-core concurrent task scheduling result data to decouple the dependencies between tasks and improve concurrent processing capabilities. Asynchronous message communication data is converted into non-blocking communication, and an asynchronous message communication queue is generated. This helps to improve the concurrent processing capability and response speed of the system. The asynchronous message communication queue is communicated in parallel pipeline data stream transmission to further improve the communication efficiency and concurrent processing capability. The performance bottleneck analysis is performed on the asynchronous parallel transmission information data, and the data is load regulated according to the analysis results. This helps to optimize the performance of the system and improve the overall processing efficiency. Therefore, the present invention improves the concurrent processing capability and overall performance of the system by performing task splitting, intelligent scheduling, asynchronous communication, and performance analysis on the cluster file client task data.

[0130] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0131] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A method for implementing multi-core concurrent load on a cluster file client, characterized in that: The following steps are involved: Step S1: Obtain cluster file client task data; Split the cluster file client task data into tasks and generate subtask sorting data; Analyze the available resources of the cluster file client and generate the available resource data of the client; Performing virtualized ultra-fine-grained task division on subtask sorting data according to client available resource data to generate ultra-fine-grained virtual task list data; Step S2: constructing a type judgment rule engine for the ultra-fine-grained virtual task list data to generate a task type judgment rule engine; prioritizing the ultra-fine-grained virtual task list data through the task type judgment rule engine to generate urgent task priority ranking data; Intelligently cache task scheduling for urgent task priority sorting data and generate multi-core concurrent task scheduling result data; Step S3: performing high-order asynchronous message communication on the multi-core concurrent task scheduling result data to generate asynchronous message communication data; performing task decoupling on the asynchronous message communication data to generate asynchronous message decoupling data; performing non-blocking communication on the asynchronous message decoupling data to generate an asynchronous message communication queue; Step S4: performing communication parallel pipeline data stream transmission on the asynchronous message communication queue to generate asynchronous parallel transmission information data; Perform performance bottleneck analysis on asynchronous parallel information transmission data to generate performance bottleneck analysis data; Through performance bottleneck analysis data, asynchronous parallel information transmission data is load-regulated to generate multi-core concurrent load adjustment data.

2. The method for implementing multi-core concurrent load of a cluster file client according to claim 1, characterized in that: Step S1 includes the following steps: Step S11: Obtain cluster file client task data; Step S12: performing task data flow relationship analysis on the cluster file client task data to obtain cluster file client relationship data; constructing a task topology structure based on the cluster file client relationship data to generate initial task topology structure data; Step S13: splitting the initial task topology data into tasks to generate subtask sequence data; sorting the subtask sequence data to generate subtask sorting data; using the initial task topology data to analyze the available resources of the cluster file client to generate client available resource data; Step S14: perform computational resource mapping on the subtask sorting data according to the client's available resource data to generate subtask resource mapping data; perform resource table allocation on the subtask resource mapping data to generate a task-resource mapping table; perform virtualized ultra-fine-grained task division on the subtask resource mapping data based on the task resource mapping table to generate ultra-fine-grained virtual task list data.

3. The method for implementing multi-core concurrent load of a cluster file client according to claim 1, characterized in that: Step S2 includes the following steps: Step S21: extracting concurrent task features from the ultra-fine-grained virtual task list data to generate concurrent task feature extraction data, wherein the concurrent task feature extraction data includes task type data, task priority data, and task dependency relationship data; Step S22: construct a type judgment rule engine based on the concurrent task feature extraction data to generate a task type judgment rule engine; classify and prioritize the ultra-fine-grained virtual task list data through the task type judgment rule engine to generate cluster file task judgment data, wherein the cluster file task judgment data includes task-intensive judgment data and IO-intensive judgment data; Step S23: Calculate the task urgency weight of the task-intensive judgment data and the IO-intensive judgment data according to the task urgency weight analysis formula to obtain the task urgency weight; prioritize the task-intensive judgment data and the IO-intensive judgment data using the task urgency weight to generate urgent task priority ranking data; Step S24: formulate an intelligent cache strategy for the priority ranking data of urgent tasks, and generate a multi-core task intelligent cache strategy; schedule the priority ranking data of urgent tasks according to the multi-core task intelligent cache strategy, and generate multi-core concurrent task scheduling result data.

4. The method for implementing multi-core concurrent load of a cluster file client according to claim 3, characterized in that: The task urgency weight analysis formula in step S23 is as follows: Where U represents the task urgency weight, P represents the task priority coefficient, R represents the task resource requirement coefficient, E represents the expected efficiency of the task, T represents the task deadline, C represents the computational amount of the task, D represents the data volume of the task, M represents the task-intensive judgment data, I represents the IO-intensive judgment data, and x represents the input variable.

5. The method for implementing multi-core concurrent load of a cluster file client according to claim 3, characterized in that: Step S24 includes the following steps: Step S241: performing task cache access feature analysis on the emergency task priority ranking data to generate task access feature data, wherein the task access feature data includes file size data, access frequency data, and most recent access time data; importing the file size data, access frequency data, and most recent access time data into a preset deep reinforcement learning model to perform learning environment modeling, and generating a reinforcement learning environment model; Step S242: Divide the task access feature data into data sets to generate a model training set and a model test set; perform model training on the reinforcement learning environment model using the model training set to generate a concurrent throughput training model; perform model testing on the concurrent throughput training model using the model test set to generate a concurrent throughput prediction model; Step S243: importing the task access feature data into the concurrent throughput prediction to generate concurrent throughput prediction data; comparing the concurrent throughput prediction data with the preset throughput threshold range, when the concurrent throughput prediction data is greater than the preset throughput threshold range, generating a first task cache strategy; when the concurrent throughput prediction data is within the preset throughput threshold range, generating a second task cache strategy; when the concurrent throughput prediction data is less than the preset throughput threshold range, generating a third task cache strategy; Step S244: when the first task cache strategy is confirmed, the cache capacity of the urgent task priority ranking data is increased to generate cache capacity increase data; the urgent task priority ranking data is cached with high frequency priority using the cache capacity increase data to generate first task cache increase data; Step S245: When the second task cache strategy is confirmed, the cache cycle analysis is performed on the emergency task priority sorting data to generate the emergency task priority cache cycle data; the emergency task priority cache cycle data is periodically cache cleaned using the LRU algorithm to generate the second task cache cleanup data; Step S246: when the third task cache strategy is confirmed, priority task access restriction is performed on the urgent task priority sorting data to generate urgent task priority access restriction data; cache pre-fetch is performed on the urgent task priority sorting data according to the urgent task priority access restriction data to generate third task cache restriction data; Step S247: Integrate the cache strategy of the first task cache increase data, the second task cache clearing data and the third task cache restriction data to generate a multi-core task intelligent cache strategy; schedule the emergency task priority sorting data according to the multi-core task intelligent cache strategy to generate multi-core concurrent task scheduling result data.

6. The method for implementing multi-core concurrent load of a cluster file client according to claim 1, characterized in that: Step S3 includes the following steps: Step S31: performing high-order asynchronous message passing framework setting on the multi-core concurrent task scheduling result data to generate a high-order asynchronous message passing framework, wherein the high-order asynchronous message passing framework includes a message producer node and a message consumer node; Step S32: transmitting the multi-core concurrent task scheduling result data to the message producer node for task message transmission to generate multi-core concurrent task transmission data; performing asynchronous message communication on the multi-core concurrent task transmission data through API calls to generate asynchronous message communication data; Step S33: using the message consumer node to perform asynchronous message processing on the asynchronous message communication data to generate asynchronous message processing data; Step S34: Decouple the asynchronous message processing data and the multi-core concurrent task scheduling result data to generate asynchronous message decoupled data; perform non-blocking communication on the asynchronous message decoupled data through the message producer node and the message consumer node to generate an asynchronous message communication queue.

7. The method for implementing multi-core concurrent load of a cluster file client according to claim 6, characterized in that: Step S33 includes the following steps: Step S331: using the message consumer node to receive the asynchronous message communication data to obtain the asynchronous message communication received data; performing data content analysis on the asynchronous message communication received data to generate message content analysis data; Step S332: performing data cleaning on the message content parsing data to generate message content cleaning data; performing data format conversion on the message content cleaning data to generate message content format conversion data; performing data standardization on the message content format conversion data by using a maximum-minimum standardization method to generate standard message content data; Step S333: updating the system status of the cluster file system based on the standard message content data to generate message content status update data; persistently storing the standard message content data according to the message content status update data to generate message content storage data; Step S334: Perform asynchronous event triggering on the message content storage data to generate a message content triggering event; perform message processing notification on the cluster file system according to the message content triggering event, thereby generating asynchronous message processing data.

8. The method for implementing multi-core concurrent load of a cluster file client according to claim 1, characterized in that: Step S4 includes the following steps: Step S41: constructing a communication parallel pipeline for the asynchronous message communication queue to generate a communication parallel pipeline; using the communication parallel pipeline to transmit data streams to the asynchronous message communication queue to generate asynchronous parallel transmission information data; Step S42: performing system performance monitoring on the asynchronous parallel transmission information data to obtain system performance monitoring data; adjusting the monitoring time period of the asynchronous parallel transmission information data according to the system performance monitoring data to generate monitoring time period data; collecting historical data of the asynchronous parallel transmission information data according to the monitoring time period data to obtain historical transmission information data; Step S43: Perform performance bottleneck analysis on the historical information transmission data to generate performance bottleneck analysis data; perform load regulation on the asynchronous parallel information transmission data through the performance bottleneck analysis data to generate multi-core concurrent load adjustment data.

9. The method for implementing multi-core concurrent load of a cluster file client according to claim 8, characterized in that: Step S41 includes the following steps: Step S411: analyzing the communication processing process of the asynchronous message communication queue to generate asynchronous message communication processing process data; dividing the pipeline process according to the asynchronous message communication processing process data to generate a communication parallel pipeline; Step S412: allocating the asynchronous message communication processing process data to parallel processing units based on the communication parallel pipeline, and generating an asynchronous message parallel task allocation scheme; Step S413: Implement the pipeline stage division diagram with parallel processing units according to the asynchronous message parallel task allocation scheme to generate parallel processing units; perform logical processing on the asynchronous message communication queue through the parallel processing units and based on the asynchronous message parallel task allocation scheme to generate asynchronous message logical data; Step S414: Perform data flow transmission on the asynchronous message logic data to generate an asynchronous message logic processing data flow; merge the processing results of the asynchronous message logic processing data flow to generate asynchronous parallel transmission information data.

10. A cluster file system, characterized in that: The method for implementing multi-core concurrent load of a cluster file client according to claim 1 is used to implement the cluster file system, and the cluster file system comprises: The task division module is used to obtain the cluster file client task data; perform task division on the cluster file client task data to generate subtask sorting data; perform available resource analysis on the cluster file client to generate client available resource data; perform virtualized ultra-fine-grained task division on the subtask sorting data according to the client available resource data to generate ultra-fine-grained virtual task list data; The judgment engine module is used to construct a type judgment rule engine for the ultra-fine-grained virtual task list data and generate a task type judgment rule engine; prioritize the ultra-fine-grained virtual task list data through the task type judgment rule engine and generate urgent task priority ranking data; perform intelligent cache task scheduling on the urgent task priority ranking data and generate multi-core concurrent task scheduling result data; The asynchronous communication module is used to perform high-order asynchronous message communication on the multi-core concurrent task scheduling result data to generate asynchronous message communication data; perform task decoupling on the asynchronous message communication data to generate asynchronous message decoupling data; perform non-blocking communication on the asynchronous message decoupling data to generate an asynchronous message communication queue; The load regulation module is used to transmit the communication parallel pipeline data stream of the asynchronous message communication queue to generate asynchronous parallel transmission information data; perform performance bottleneck analysis on the asynchronous parallel transmission information data to generate performance bottleneck analysis data; perform load regulation on the asynchronous parallel transmission information data through the performance bottleneck analysis data to generate multi-core concurrent load adjustment data.

Images