Distributed computing system based on pre-scheduling resource collaborative adaptation

Abstract

The application provides a distributed computing system based on pre-scheduling resource cooperative adaptation, and relates to the technical field of resource cooperative allocation system. The distributed computing system based on pre-scheduling resource cooperative adaptation aims to solve the problems of inflexible resource scheduling, lack of coordination between application configuration and resource scheduling and the like in the prior art. By introducing a dynamic scheduling mechanism of resources, the application realizes the organic combination of resource pre-scheduling and flexible allocation. Meanwhile, by the cooperation between application configuration and resource scheduling, the application realizes the optimized allocation and execution of computing tasks. The system is composed of a server and a client, wherein the client has pre-scheduling resources, is used for feeding back resource states in real time, receiving and executing computing tasks, and the server is responsible for monitoring, scheduling and task distribution of global resources. Through the designed heuristic cooperative adaptation algorithm, the server can dynamically schedule resources in the pre-scheduling resources, and ensure the efficiency and stability of task execution.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of resource cooperative allocation systems, and specifically to a distributed computing system based on pre-scheduling resource cooperative adaptation. BACKGROUND

[0002] With the explosive growth of data volume and the increasing complexity of computing requirements and computing tasks, traditional centralized computing models face bottlenecks and challenges. Distributed computing, as a solution, can handle large-scale data sets and complex computing tasks by utilizing multiple computing resources, including cloud computing, edge computing, and other distributed architectures. In addition, distributed computing supports parallel processing, allowing multiple tasks to be processed simultaneously, improving the utilization of computing resources. This is of great significance in the fields of large-scale data analysis, artificial intelligence, and autonomous driving. In particular, edge computing, as an important part of distributed computing, has emerged in recent years.

[0003] Edge computing pushes computing resources and data processing capabilities to the network edge, enabling faster data processing and real-time response. By deploying edge servers and computing nodes closer to data sources and terminal devices, edge computing effectively reduces data transmission delay and network congestion, improving the overall performance and efficiency of the system. For example, in the application of vehicle-machine collaboration, the information obtained by intelligent vehicles through cameras and sensors needs to be processed and calculated quickly. Servers and large computing nodes that are far away cannot meet the low-latency requirements, often have large transmission delays and queuing delays, and the vehicle's own computing devices are not enough to support. At this time, a part of the computing tasks can be distributed to idle devices such as mobile phones, laptops, and other computing devices connected to the Internet of Vehicles through a distributed system, thereby reducing the burden on the vehicle's computing devices and improving the performance and response speed of the entire system. In addition, through wireless communication between mobile devices and vehicle systems, real-time data transmission and interaction can be achieved, allowing vehicles to quickly obtain and process information and improving the stability and safety of autonomous driving systems.

[0004] To fully and safely utilize the excess resources of computing devices, it is necessary to use container virtualization technology. Nowadays, most enterprises and users are turning to multi-cloud infrastructure to reduce the risk of over-reliance on a single vendor and to obtain the best services and resources for performance optimization. Virtualization is one of the key technologies of distributed computing and cloud computing, and most cloud-based systems are based on virtualization [1]. Docker is a container-based virtualization technology used to develop various types of multi-cloud distributed systems. It divides the resources of a physical server into multiple virtual machine instances, enabling the isolation and management of resources, and allowing multiple applications to run simultaneously on the same physical device. This flexibility allows the system to dynamically adjust resource configurations based on changes in workloads, improving the system's scalability and performance. In addition, virtualization technology provides fault tolerance and high availability for the system by quickly migrating virtual machines to handle physical server failures, ensuring continuous and stable operation of the system. At the same time, virtualization improves the utilization of hardware resources, reduces costs, and simplifies the deployment and management of systems. In summary, virtualization technology provides fundamental support for distributed computing and ensures the reliability and efficiency of the system. However, Docker is mainly used for software development, and less attention is paid to resource control and data processing based on containers.

[0005] In addition to virtualization technology, edge computing also requires a large computing platform and communication base, as a large number of computing tasks and data processing requirements are involved in the edge computing environment. A centralized platform is essential to receive, distribute tasks to multiple computing nodes, and manage and optimize resources. This platform can effectively schedule and allocate tasks to ensure timely completion and unified management and optimization of resources, including resource scheduling, load balancing, etc. In addition, this platform can receive, integrate, and analyze data generated in the edge computing environment to support decision-making and applications. Therefore, the computing platform plays a crucial role in edge computing and can drive the development and application of edge computing technology.

[0006] Edge computing provides a powerful solution to current computing needs, but also faces some challenges. One of the most prominent issues is the complexity of resource management faced by large computing platforms. In a distributed environment, the heterogeneity of resources among nodes, network latency, and other factors make the effective use of resources more complex. For example, in a distributed computing network, some nodes may be equipped with high-performance processors and large memory, while others may only provide lower-performance hardware. In addition, challenges in task scheduling, load balancing, and data transmission also limit the performance and efficiency of distributed systems.

[0007] To address these issues, dynamic scheduling of resources has become an attractive concept. By monitoring and analyzing the resource conditions of computing nodes in real time, the system can intelligently adjust the allocation strategy of resources, so that each node can fully utilize its resources, while avoiding overload or resource idling, improving system resource utilization efficiency. SUMMARY

[0008] In view of the deficiencies of the prior art, the present application provides a distributed computing system based on pre-scheduled resource cooperative adaptation, wherein the pre-scheduled resource module and the information storage module are implemented based on Docker containers and Redis databases. The system is a research and design of a resource monitoring and control module, which provides services and support for the resource scheduling function of the OpenSAI platform. OpenSAI is a general distributed network artificial intelligence platform that can unify the elastic algorithm-storage-network scheduling of end-to-end systems, provide service customization networks, and realize the cooperative bidirectional optimization of network and application computing, i.e., the cooperative configuration and online adaptation of algorithm-network resources and application execution, and support dynamic offloading of computing tasks, ensuring the execution efficiency and reliability of application services. On the one hand, the scheduling mechanism combining resource pre-scheduling and flexible allocation improves the adaptability and flexibility of resource scheduling. On the other hand, the cooperative optimization of application configuration and resource scheduling solves the problems of isolation and inefficiency in traditional scheduling methods.

[0009] To achieve the above object, the present application is implemented by the following technical scheme: a distributed computing system based on pre-scheduled resource cooperative adaptation, specifically comprising the following steps:

[0010] S1. System design

[0011] S101. The server needs to obtain part of the resource information of the client and store these information in the database. These resource information includes the CPU, GPU, network and memory usage of the client, which is used for subsequent dynamic resource scheduling and monitoring;

[0012] S102. The server needs to build multiple clients and pre-schedule the required resources for quick deployment and use;

[0013] S103. The server is responsible for sending different tasks and parameter settings to different clients. Through communication with the client, the server can send task files to the corresponding client according to the needs of the system, so as to perform calculation and processing;

[0014] S104. The server also needs to monitor the running state and resource utilization of the system, and perform dynamic resource scheduling according to the dynamic scheduling algorithm, so as to realize the efficiency improvement and performance optimization of the system;

[0015] S105. The server needs to receive the calculation results and integrate and store them. Once the client completes the calculation task and sends the calculation results, the result type message will be received by the server, and the server will store the corresponding calculation results and time information in the system for subsequent data analysis and processing.

[0016] S2. Collaborative adaptive algorithm design

[0017] S201. When building a pre-scheduled resource client, the server needs to pre-configure certain resources according to the system's needs and pre-scheduled algorithm rules, in order to be quickly deployed and used;

[0018] S202. The server also needs to dynamically schedule resources for the client according to the dynamic scheduling algorithm. By monitoring the system's running state and resource utilization, the server can dynamically schedule resources to improve the efficiency and performance of the system;

[0019] S203. For the application side, application scheduling dynamically adjusts the task configuration parameters of the application based on the running state of the monitoring system and the performance indicators of the application, optimizing the running efficiency and user experience of the application. Application scheduling can dynamically adjust the number of application instances, task priorities and task allocation strategies according to the load. By monitoring the health status and performance of the application in real time, application scheduling can automatically recover failed instances, optimize traffic distribution, and quickly respond to changes in user demand, ultimately improving the overall efficiency of the system and optimizing service performance. Cooperate with resource scheduling to achieve collaborative adaptation of the system.

[0020] S204. The server needs to receive the calculation results and integrate and store them. Once the client completes the calculation task and sends the calculation results, the result type message will be received by the server, and the server will store the corresponding calculation results and time information in the system for subsequent data analysis and processing.

[0021] Preferably, in S202, a priority-based greedy algorithm is designed, and the server obtains the state information of the client every 1 second. For each client, the server uses the obtained state information, including CPU and memory usage, to process the sequence using the Exponential Weighted Moving Average (EWMA) algorithm, and then slide window average as the current pre-scheduled resource state priority evaluation index. The EWMA algorithm is as follows:

[0022]

[0023] Where EMA is the exponential weighted moving average value at each time, X is the observation value, and a is the smoothing factor, which controls the weight of the data points. The closer the value is to 1, the greater the weight of the latest data points.

[0024] Preferably, according to the rules set in advance, the server divides the resource state of the current pre-scheduled resource into three different states: idle, available and full load, and the window size and state threshold can be adjusted as parameters to meet the needs in different scenarios. After the priority setting is completed, the allocation cycle process is as follows: first, traverse from the highest priority client downward, if the client is not in the idle state, it needs to obtain resources from the lower priority client, in this case, it needs to traverse from the lowest priority client upward, if the client is in the available or idle state, and the low priority client is still higher than the minimum resource limit to ensure the normal operation of the client after giving the resource, the resource is given, and the client resource configuration list is updated; otherwise, the resource is not given, after completing the two-layer loop, all clients are traversed, and all clients are updated according to the updated client resource configuration list, then, after waiting for a certain time window, the next allocation is performed, and the cycle continues until all the computing tasks of the clients are completed.

[0025] Preferably, the application dynamic scheduling is based on the running state of the monitoring system, the application performance index, and the future load trend predicted by the neural network, and dynamically adjusts the number of application instances, task priority and task allocation strategy according to the real-time monitoring and prediction results, while ensuring the normal migration and operation of the task when the underlying resource changes. At the same time, by monitoring the health state and performance of the application in real time, and combining the neural network model to predict possible bottlenecks and resource requirements, the application scheduling can deploy resources in advance, automatically recover failed instances, optimize traffic distribution, and quickly respond when user demand changes, ultimately improving the overall efficiency of the system and optimizing the service performance.

[0026] Preferably, a cooperative adaptation mechanism combining resource pre-scheduling and flexible allocation is realized: the resource scheduling mechanism proposed by the application can be applied to resources, storage and network resources, and by combining resource static pre-scheduling and dynamic scheduling, the efficient use of resources during task execution is ensured, and the adaptability and flexibility of the system are improved.

[0027] Preferably, an optimization mechanism of application configuration and resource scheduling is realized: the cooperative optimization between application configuration and resource scheduling is realized, so that the resource scheduling is more in line with the actual task demand, avoiding the scheduling isolation in the traditional method, and improving the overall efficiency of the system.

[0028] Preferably, a real-time resource monitoring and scheduling feedback mechanism based on containers is realized: in the containerized environment, the client provides part of the resource information in real time, and transmits these information back to the server, and the server can dynamically adjust the scheduling strategy of the resource based on the real-time feedback, realizing the fine management of the resource, see the system design and flowchart part in the specific embodiment.

[0029] Preferably, in evaluating the system performance, four evaluation indexes are mainly used, resource utilization efficiency, computing delay, reliability index, and load balancing index.

[0030] Preferably, the resource utilization efficiency refers to the product of the number of CPU cores and time, which reflects the resources used by the system in executing the computing task, the sampling period is 1 second, and the overall resource efficiency of the system is calculated by the following formula:

[0031]

[0032] Wherein N refers to the number of CPU cores, and C refers to the CPU usage.

[0033] Preferably, the computing delay refers to the time length of the server receiving all task results, which depends on the longest completion time of the task.

[0034] Preferably, the load balancing index refers to the standard deviation of the task completion time of each client under the same task size, and the smaller the variance, the closer the task execution time of each client, and the more balanced the load.

[0035] Preferably, the reliability index refers to the task completion rate within a given time, that is, the amount of data processing that has been completed and the total amount of data processing, and the higher the completion rate, the higher the system reliability can be considered.

[0036] The application provides a distributed computing system based on pre-scheduling resource collaborative adaptation.

[0037] 1. The application provides a distributed computing system based on pre-scheduling resource collaborative adaptation, which combines resource pre-scheduling and flexible allocation, so that the system can dynamically adjust resource scheduling according to the changes of task load, avoid resource waste, ensure the resource needs of high-priority tasks, and improve the flexibility of resource scheduling. This flexibility significantly improves resource utilization and reduces the cost of idle resources.

[0038] 2. The application provides a distributed computing system based on pre-scheduling resource collaborative adaptation, which cooperatively optimizes application configuration and resource scheduling, so that the system can reasonably schedule resources according to the actual needs of the application, avoid the low efficiency and resource conflict problems caused by the separation of the two in the traditional scheduling mode, and enhance the stability and operating efficiency of the overall system.

[0039] 3. The application provides a distributed computing system based on pre-scheduling resource collaborative adaptation, which can dynamically adjust the allocation and execution strategy of tasks by real-time acquisition of client resource information, ensure efficient completion of computing tasks, and reasonably allocate tasks when resources are tight. This not only improves the execution speed of tasks, but also reduces the waiting time of tasks in the queue, and improves the efficiency of task execution.

[0040] 4. The application provides a distributed computing system based on pre-scheduling resource collaborative adaptation, which can flexibly schedule resources according to different task types and needs through flexible resource scheduling and heuristic collaborative adaptation algorithm, realizes the elastic expansion of the system, can cope with sudden computing needs, improves the load processing capacity of the system, and enhances the stability of the system.

[0041] 5. The application provides a distributed computing system based on pre-scheduling resource collaborative adaptation, which can accurately manage and allocate resources by providing part of the resource information of the client in real time, improve the control ability of the system over computing resources, avoid resource scheduling errors caused by information lag in traditional ways, and make the visualization of computing resources convenient for accurate management.

[0042] 6. The application provides a distributed computing system based on pre-scheduling resource collaborative adaptation, which provides real-time display of computing progress and feedback function of computing results, so that the system can monitor the execution state of the task in real time, timely adjust the allocation strategy, further improve the reliability and controllability of task execution, and realize real-time feedback and monitoring of task execution state.

[0043] 7. The application provides a distributed computing system based on pre-scheduling resource collaborative adaptation, which effectively reduces the idle and waste of computing resources through dynamic scheduling and collaborative allocation mechanism, reduces the hardware cost and energy consumption, and reduces the dependence on manual intervention, thereby reducing the operation and maintenance cost of the system. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a connection and communication flow diagram of the server and the client of the application;

[0045] Figure 2 It is a pseudo code diagram of the dynamic scheduling algorithm based on priority of the application;

[0046] Figure 3 It is a horizontal comparison structure diagram of the resource scheduling algorithm of the application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0048] Embodiment 1

[0049] The embodiment of the application provides a distributed computing system based on pre-scheduling resource cooperative adaptation, as shown in the figure Figure 1 As shown in the figure, the server mainly includes two threads-information processing and cooperative adaptation algorithm, and the specific work flow is as follows:

[0050] S1. System design

[0051] S101. The server needs to obtain part of the resource information of the client and store the information into the Redis database, the Redis database is a key-value pair storage system based on memory, has the characteristics of fast reading and writing, efficient storage. It is often used for real-time data caching and storage. In the application, Redis is used to store real-time client information to help the system make quick decisions and resource scheduling. These resource information includes CPU usage, GPU usage, memory usage of the client, which is used for subsequent dynamic resource scheduling and monitoring;

[0052] S102. The server needs to build the Docker container of the pre-scheduling resource, the Docker container is a lightweight virtualization technology, which is used for running application programs in a computing environment, each container includes application and its dependent library, configuration, etc., to ensure that the application can maintain consistent running environment on any platform. Compared with traditional virtual machine, Docker container has the advantages of efficiency and resource utilization, according to the demand of the system and the algorithm rules of pre-scheduling, the server will pre-configure a certain number of Docker container resources for quick deployment and use;

[0053] S103. The server is responsible for sending different tasks and parameter settings to different clients, through the communication between the server and the client, the server can send task files to the corresponding client according to the demand of the system, so as to carry out calculation and processing;

[0054] S104. The server also needs to monitor the running state and resource utilization of the system, and carry out dynamic resource scheduling of the container according to the dynamic scheduling algorithm, so as to realize the efficiency improvement and performance optimization of the system;

[0055] S105. The server needs to receive the calculation result and store it, once the client completes the calculation task and sends the calculation result, the result type message will be received by the server, and the corresponding calculation result and time information will be stored in the system for subsequent data analysis and processing.

[0056] Specifically, the client mainly implements four functions: providing real-time pre-scheduling resource information, receiving computing tasks, executing computing tasks and displaying computing progress, and returning computing results. The client first sends a "handshake" message to the server to indicate that it is online, and then receives the computing tasks and related parameters and data sent by the server. Then, the client uses double threads for processing: one thread is responsible for running the computing task using the given parameters, and displays the computing progress and estimated task completion time in real time by using the tqdm library; the other thread is used to calculate the real-time delay information and send the result to the server. Until all threads complete the computing task, the client sends the result to the server after integration. This allows the client to efficiently receive tasks, execute calculations, and communicate with the server in real time, thereby achieving smooth execution of tasks and timely return of results.

[0057] S2. Collaborative adaptation algorithm design

[0058] Heuristic algorithm is a problem-solving algorithm based on empirical rules and heuristic information, commonly used to solve complex optimization problems. The heuristic collaborative adaptation algorithm combines the characteristics and resource requirements of different tasks, intelligently allocates appropriate resources to tasks, and improves resource utilization efficiency and system performance.

[0059] S201. When building a client for pre-scheduling resources, the server needs to pre-configure certain resources according to the system's needs and the algorithm rules of pre-scheduling, in order to be quickly deployed and used;

[0060] S202. The server also needs to dynamically schedule containers based on dynamic scheduling algorithms. By monitoring the system's running state and resource utilization, the server can dynamically schedule container resources to improve system efficiency and optimize performance;

[0061] S203. For the application side, application scheduling dynamically adjusts the task configuration parameters of the application based on the monitoring of the system's running state and application performance indicators, optimizing the running efficiency of the application and user experience. Application scheduling can dynamically adjust the number of application instances, task priority and task allocation strategy based on load. By monitoring the health status and performance of the application in real time, application scheduling can automatically recover failed instances, optimize traffic distribution, and quickly respond to changes in user demand, ultimately improving the overall efficiency of the system and optimizing service performance. Cooperating with resource scheduling, the system achieves collaborative adaptation.

[0062] S204. The server needs to receive and integrate the computing results. Once the client completes the computing task and sends the computing result, the result type message will be received by the server, which will store the corresponding computing result and time information in the system for subsequent data analysis and processing.

[0063] Preferably, in S202, a priority-based greedy algorithm is designed, and the server obtains the state information of the client every 1 second. For each client, the server uses the obtained state information, including the CPU and memory usage, to evaluate the current pre-scheduling resource state priority using the Exponential Weighted Moving Average (EWMA) algorithm after processing the sequence and sliding window averaging. The EWMA algorithm is as follows:

[0064]

[0065] where EMA is the exponential weighted moving average value at each time, X is the observation value, and a is the smoothing factor that controls the weight of the data points. The closer the value of a is to 1, the greater the weight of the latest data points.

[0066] Specifically, the algorithm sets the weight of the data to decay exponentially over time, with the latest data points having higher weights and the older data points having lower weights. The EWMA algorithm is commonly used to smooth time series data to better capture the trends and changes in the data while reducing the impact of noise.

[0067] As shown in the accompanying Figure 2 According to the pre-set rules, the server divides the current pre-scheduling resource state into three different states: Free (marked as -1), Available (marked as 0), and Full (marked as 1). The window size and state threshold values can be adjusted as parameters to meet the needs of different scenarios. After the priority is set, the allocation algorithm follows the following allocation cycle process: First, traverse from the highest priority client downward. If the client is not in the idle state, it needs to obtain resources from lower priority clients. In this case, traverse from the lowest priority client upward. If the client is in the available or idle state, and the low-priority client is still higher than the minimum resource limit to ensure the normal operation of the pre-scheduling resource after giving out the resource, then give out the resource and update the pre-scheduling resource configuration list. Otherwise, do not give out the resource. After completing the two-layer loop, traverse all clients and update the resource scheduling of all clients according to the updated pre-scheduling resource configuration list. Then, wait for a certain time window and perform the next allocation. Repeat the process until all client computing tasks are completed.

[0068] Embodiment 2

[0069] The embodiment of the application provides a distributed computing system based on pre-scheduling resource cooperative adaptation, which is applicable to multiple technical fields, mainly including:

[0070] 1) Cloud computing and edge computing: The dynamic resource scheduling mechanism in this invention is very suitable for application in cloud computing platform, which can efficiently manage the computing resources in cloud server and ensure the smooth execution of tasks in multi-tenant environment. At the same time, in the edge computing scenario, the resource utilization of edge devices can also be optimized through pre-scheduling resource technology to improve task execution efficiency and system response speed.

[0071] 2) Data processing and distributed computing: The scheduling mechanism can be used to optimize the resource scheduling of computing tasks, improve the resource utilization in computing cluster, and ensure the efficient processing of large-scale computing tasks and the optimization of data transmission.

[0072] 3) Artificial intelligence and machine learning: In the artificial intelligence training task, model training often requires a large amount of computing resources, and the collaborative allocation mechanism can dynamically schedule resources according to the demand of training task, ensure efficient use of resources, shorten training time, and adapt to different scale and load of computing demand.

[0073] 4) Pre-scheduling resource application deployment and microservices architecture: Through the support of pre-scheduling resource technology such as Docker, the efficient scheduling and management of application resources are realized. This has significant advantages for dynamic expansion, resource optimization allocation and other problems under microservices architecture, which can support the flexible deployment and expansion of microservices in multiple nodes.

[0074] Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A distributed computing system based on pre-scheduled resource collaborative adaptation, characterized in that, Specifically comprising the following steps: S1. System design S101. The server needs to obtain part of the resource information of the pre-scheduled resource client and store these information in the database; these resource information includes the CPU, GPU, network and memory usage of the client, which is used for subsequent dynamic resource scheduling and monitoring; S102. The server needs to build multiple clients and pre-schedule the required resources for quick deployment and use; S103. The server is responsible for sending different tasks and parameter settings to different clients. Through communication with the client, the server can send application task files to the corresponding client according to the needs of the system, so as to perform calculation and processing; S104. The server also needs to monitor the running state and resource utilization of the system, and dynamically schedule resources according to the collaborative adaptation algorithm, so as to realize the efficiency improvement and performance optimization of the system; S105. The server needs to receive and integrate the calculation results, and once the client completes the calculation task and sends the calculation results, the result type message will be received by the server, and the corresponding calculation results and time information will be stored in the system for subsequent data analysis and processing; S2. Collaborative adaptation algorithm design S201. When building pre-scheduled resource clients, the server needs to pre-configure certain resources according to the needs of the system and the pre-scheduling algorithm rules for quick deployment and use; S202. The server also needs to dynamically schedule resources for pre-scheduled resources according to the dynamic scheduling algorithm; by monitoring the running state and resource utilization of the system, the server can dynamically schedule resources to realize the efficiency improvement and performance optimization of the system; S203. For the application side, the application configuration dynamically adjusts the task allocation parameters of the application based on the monitoring of the running state and application performance indicators of the system, and optimizes the running efficiency and user experience of the application; Application scheduling can dynamically adjust the number of application instances, task priority and task allocation strategy according to the load; Through real-time monitoring of the health state and performance of the application, the application scheduling can automatically recover the failed instances, optimize the traffic distribution, and quickly respond to changes in user demand, ultimately realizing the improvement of the overall efficiency of the system and the optimization of service performance; Cooperate with resource scheduling to realize the collaborative adaptation of the system; S204. The server needs to receive and integrate the calculation results, and once the client completes the calculation task and sends the calculation results, the result type message will be received by the server, and the corresponding calculation results and time information will be stored in the system for subsequent data analysis and processing.

2. The distributed computing system based on pre-scheduling resource collaborative adaptation according to claim 1, characterized in that: In S202, a priority-based greedy algorithm is designed, and the server obtains the state information of the client every 1 second; for each client, the server uses the obtained state information, including the CPU and memory usage, and then uses the exponential weighted moving average (EWMA) algorithm to process the sequence and then slide window average as the evaluation index of the current pre-scheduled resource state priority; The EWMA algorithm is specifically as follows: Wherein EMA is the exponential moving average value of each time, X is the observation value, and alpha is a smoothing factor that controls the weight of data points, and the closer the value is to 1, the greater the weight of the latest data point. 3.The distributed computing system based on pre-scheduling resource collaborative adaptation according to claim 2, characterized in that: According to the rules set in advance, the server divides the resource state of the current client into three different states: idle, available and full load, and the window size and state threshold can be adjusted as parameters to meet the needs of different scenarios. After the priority setting is completed, the allocation algorithm has the following allocation cycle process: first, traverse from the client with the highest priority downward, if the client is not in the idle state, it needs to obtain resources from the client with lower priority, in this case, it needs to traverse from the client with the lowest priority upward, if the client is in the available or idle state, and the low-priority client is still higher than the minimum resource limit to ensure the normal operation of the client after giving the resource, the resource is given, and the client resource configuration list is updated; otherwise, the resource is not given, after completing the two cycles, all clients are traversed, and the resource scheduling of all clients is updated according to the updated client resource configuration list.

4. The distributed computing system based on pre-scheduling resource collaborative adaptation according to claim 1, characterized in that: The application dynamic scheduling is based on the running state of the monitoring system, the application performance index, and the future load trend predicted by the neural network, dynamically adjusts the number of application instances, task priority and task allocation strategy according to the real-time monitoring and prediction results, and ensures the normal migration and operation of the task when the underlying resource changes; at the same time, through real-time monitoring of the health state and performance of the application, and combining the neural network model to predict possible bottlenecks and resource requirements, the application scheduling can deploy resources in advance, optimize traffic distribution, and quickly respond when the user demand changes, finally realizes the improvement of the overall efficiency of the system and the optimization of the service performance.

5. The distributed computing system based on pre-scheduling resource collaborative adaptation according to claim 1, characterized in that: A collaborative adaptation mechanism combining resource pre-scheduling and flexible allocation is realized: the resource scheduling mechanism proposed by the application can be applied to computing power resources, storage and network resources, by combining static pre-scheduling and dynamic scheduling of resources, ensuring efficient use of resources during task execution, and improving the adaptability and flexibility of the system.

6. The distributed computing system based on pre-scheduling resource collaborative adaptation according to claim 1, characterized in that: An optimization mechanism of application configuration and resource scheduling is realized: the collaborative optimization between application configuration and resource scheduling is realized, so that the resource scheduling is more in line with the actual task demand, avoiding the scheduling isolation in the traditional method, and improving the overall efficiency of the system.

7. The distributed computing system based on pre-scheduling resource collaborative adaptation according to claim 1, characterized in that: A real-time resource monitoring and scheduling feedback mechanism based on containers is realized: by providing part of the resource information by the client in a containerized environment and feeding it back to the server, the server can dynamically adjust the scheduling strategy of the resource based on real-time feedback, realizing fine management of the resource. A collaborative adaptation mechanism combining resource pre-scheduling and flexible allocation is realized: the resource scheduling mechanism proposed by the application can be applied to computing power resources, storage and network resources, by combining static pre-scheduling and dynamic scheduling of resources, ensuring efficient use of resources during task execution, and improving the adaptability and flexibility of the system. An optimization mechanism of application configuration and resource scheduling is realized: the collaborative optimization between application configuration and resource scheduling is realized, so that the resource scheduling is more in line with the actual task demand, avoiding the scheduling isolation in the traditional method, and improving the overall efficiency of the system. A real-time resource monitoring and scheduling feedback mechanism based on containers is realized: by providing part of the resource information by the client in a containerized environment and feeding it back to the server, the server can dynamically adjust the scheduling strategy of the resource based on real-time feedback, realizing fine management of the resource.

Images