Serverless distributed computing platform for big data scene

Abstract

The invention provides a distributed computing platform which is oriented to a big data scene and is based on a Serverless framework. The platform comprises an elastic computing resource pool composed of multiple nodes, and computing power resources can be supplied as required by combining a dual capacity expansion and contraction mechanism and intelligent resource analysis scheduling. The platform adopts a Serverless task operation model, simplifies big data application deployment, and realizes performance isolation in a multi-tenant scene through uniform resource management. In addition, the visual monitoring platform is assisted by intelligent analysis, and decision support is provided for operation and maintenance management. Compared with traditional static resource scheduling, the method has the advantages that the resource utilization rate can be remarkably increased, the operation and maintenance complexity is reduced, powerful resource isolation and fairness guarantee are provided in a multi-tenant environment, and technical innovation and application expansion of distributed computing in the big data era are effectively supported.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing technology, and in particular to a distributed computing platform based on a Serverless architecture for big data computing scenarios. Background Art

[0002] In today's big data era, enterprise-level data processing tasks present huge resource demand fluctuations and computing power isolation challenges. Traditional data processing platforms mainly rely on static resource allocation and centralized task scheduling mechanisms, which are difficult to adapt to the rapid growth of data volume and complex and changing business needs. Efficient and flexible distributed computing platforms have important application value in the field of big data processing.

[0003] At present, distributed clusters based on big data frameworks such as Hadoop and Spark are mainly used to process massive data. These frameworks improve data processing efficiency to a certain extent by distributing computing tasks to multiple nodes for parallel execution. However, traditional big data processing platforms usually adopt a static resource allocation strategy, that is, pre-allocating fixed computing resources for each task. This approach is difficult to cope with dynamically changing load requirements, and is prone to waste of resources or task blocking, limiting the system's elasticity and service quality.

[0004] In addition, most existing distributed computing platforms lack effective multi-tenant resource isolation mechanisms. In a shared cluster environment, tasks of different users or applications are prone to mutual interference and resource preemption, affecting the performance of key tasks. At the same time, users need to manually manage the underlying computing resources, which increases the complexity of application development and operation and maintenance. These problems restrict the flexibility, reliability and ease of use of big data processing platforms.

[0005] In order to overcome the above difficulties, the industry has tried to introduce the concept of serverless computing, making the task running environment serverless and realizing on-demand supply and automatic scaling of computing resources. Serverless platforms such as AWS Lambda and Google Cloud Functions have made certain progress in scenarios such as event-driven and function computing. However, these general serverless platforms still face many challenges when supporting complex big data computing tasks, such as low resource utilization and high cold start latency. How to further optimize resource management and task scheduling strategies based on the serverless architecture has become a research hotspot in the field of big data computing.

[0006] In summary, traditional big data processing platforms still have deficiencies in resource utilization efficiency, multi-tenant isolation, and application development and operation efficiency. In order to cope with the growing data processing needs and business challenges, it is urgent to develop a flexible and efficient distributed computing platform. The ideal platform should be able to provide dynamic and elastic resource supply capabilities, intelligent and optimized resource allocation and scheduling mechanisms, and a highly isolated multi-tenant operating environment. This will help significantly improve the overall performance of the system, reduce application delivery costs, and promote the innovative development of big data processing technology. Summary of the invention

[0007] The purpose of the present invention is to solve the deficiencies of existing big data processing platforms in terms of resource utilization efficiency, multi-tenant isolation, and application development and operation efficiency. By introducing the Serverless architecture and innovative resource management mechanism, a highly elastic, isolated, and easy-to-use distributed computing platform is implemented to meet the growing data processing needs. In addition, the present invention also aims to reduce the user's usage threshold and application delivery costs, and promote the popularization and application of big data processing technology.

[0008] To achieve the above objectives, the present invention provides a distributed computing platform based on Serverless architecture for big data computing scenarios. Its core feature is that the platform includes an elastic resource pool composed of multiple computing nodes, combined with a dual expansion and contraction mechanism and an intelligent resource analysis and scheduling module, which can automatically adjust the supply of computing resources according to load requirements. At the same time, the platform adopts a Serverless task operation model, and users do not need to pay attention to the management of underlying resources, reducing the complexity of application development and deployment. In a multi-tenant scenario, the platform ensures the performance stability of key tasks through a unified resource management and isolation mechanism.

[0009] Specifically, the elastic computing resource pool consists of multiple computing nodes, each of which is virtualized into multiple lightweight computing units, such as containers, to support fine-grained resource allocation and scheduling. The dual scaling mechanism responds to severe load fluctuations by horizontally scaling the entire resource pool, and vertically scaling a single computing unit to meet the dynamic resource requirements of the started tasks.

[0010] Furthermore, the intelligent resource analysis and scheduling module is based on machine learning algorithms, including but not limited to decision trees, support vector machines, neural networks, etc., which continuously analyze the resource consumption patterns of different tasks, predict their future resource needs, and adjust the resource allocation strategy in advance accordingly. This predictive scheduling method can maximize resource utilization while ensuring task SLA and avoid waste caused by over-configuration.

[0011] Preferably, the platform adopts the Serverless task operation model, and users only need to submit task code and configuration without worrying about the supply and management of underlying computing resources. The system automatically selects the optimal computing resource combination according to the real-time load of the task, and adopts an on-demand billing method, which reduces the user's operation and maintenance burden and usage costs.

[0012] In some embodiments, the platform also includes a scheduling policy configuration module, through which administrators can flexibly set resource allocation policies and task priority rules according to business needs, so as to intervene in and optimize the system's scheduling behavior in specific scenarios.

[0013] In one embodiment, the platform also introduces a fine-grained multi-tenant resource isolation mechanism. By setting strict resource quotas and access controls between different tenants, it ensures that key tasks are not interfered with by other tasks, providing predictable performance guarantees.

[0014] In some embodiments, the platform also provides a visual monitoring and management interface. Administrators can view the system's operating status in real time, such as key indicators such as node load and task distribution. The platform also has built-in anomaly detection and root cause analysis functions based on machine learning to help administrators quickly discover and locate potential problems.

[0015] In addition, the present invention can also be seamlessly integrated with mainstream big data processing frameworks and tools. Users can smoothly migrate existing Spark, Flink and other tasks to the Serverless environment without modifying the code, taking into account both ease of use and scalability.

[0016] In a preferred implementation, the platform adopts a loosely coupled microservice architecture, where different functional components communicate through standard API interfaces, making it easy to expand and customize. The platform also supports multi-cloud and hybrid cloud deployment modes, and users can flexibly choose according to security, cost and other requirements.

[0017] In another preferred embodiment, the platform also integrates functions such as data privacy protection and security auditing to ensure the compliance of sensitive data during the processing process and meet the stringent requirements of enterprise-level users.

[0018] By adopting the above scheme, the present invention has the following beneficial effects: (1) It provides highly flexible resource supply capabilities to cope with dynamically changing data processing loads, improving resource utilization and task response speed; (2) It achieves fine-grained, comprehensive multi-tenant resource isolation, ensuring the performance stability and data security of key tasks; (3) Simplified the development and deployment process of big data applications, lowered the user threshold, and accelerated product iteration and business innovation; (4) Optimized the allocation and scheduling strategies of computing resources, minimizing cost investment while meeting task SLAs, saving expenses for enterprises; (5) It integrates intelligent monitoring, analysis, tuning and other auxiliary functions, greatly improving the system's operation and maintenance management efficiency.

[0019] In summary, the present invention has created a new distributed computing model that is flexible, efficient, secure, reliable, simple and easy to use, which has effectively promoted the integration and innovation of Serverless and big data technology, and provided a new data processing solution for enterprise-level users. It has broad application prospects and significant commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is a schematic diagram of the overall architecture of the distributed computing platform of the present invention.

[0022] Figure 2 This is a schematic diagram of the elastic resource pool structure.

[0023] Figure 3 The workflow diagram of the dual scaling mechanism is shown.

[0024] Figure 4 The inner working principle of the intelligent resource scheduling module is explained.

[0025] Figure 5 This shows the timing flow chart of a Serverless task from submission to completion.

[0026] Figure 6 Explains the basic principles of multi-tenant resource isolation.

[0027] Figure 7 Shows the interface layout of the visual monitoring management module.

[0028] Figure 8 Demonstrated heterogeneous fusion cluster architecture.

[0029] Fig. 9 It is a workflow scheduling diagram. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] In the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0032] Example 1: This example provides a distributed computing platform based on Serverless architecture for big data computing scenarios. Figure 1 As shown in the figure, the core components of the platform include: elastic computing resource pool, dual expansion and contraction management module, intelligent resource analysis and scheduling module, Serverless task running environment, multi-tenant resource management module and visual monitoring management module.

[0033] The elastic computing resource pool is the physical infrastructure layer of the entire platform. Figure 2 As shown in the figure, the resource pool adopts a two-level resource abstraction model. Through the hierarchical management of server nodes and lightweight computing units, coarse-grained elastic scaling is achieved at the node level, and fine-grained resource allocation and isolation is supported at the unit level.

[0034] Specifically, the server nodes in the resource pool can be dynamically added or deleted according to the load conditions, providing the first level of elasticity. Each server node can be further divided into multiple computing units in the form of containers or micro VMs, which are the basic units of resource allocation and task scheduling. This hierarchical abstract model can match computing requirements of different granularities while improving resource utilization, thereby achieving a balance between performance and cost. Compared with traditional static resource allocation schemes, the elastic resource pool of the present invention has stronger adaptability and flexibility.

[0035] Based on the elastic resource pool, the present invention innovatively designs a dual expansion and contraction management mechanism in the form of Serverless to cope with the drastic load fluctuations of big data computing tasks.

[0036] refer to Figure 3The mechanism mainly includes two levels: one is to achieve horizontal scaling by adding or reducing nodes at the resource pool level, and the other is to achieve vertical scaling by adjusting resource quotas at the computing unit level. The two work together to cover load changes from milliseconds to hours.

[0037] Among them, horizontal scaling mainly copes with long-term and large-scale load changes, while vertical scaling focuses on quickly responding to the dynamic resource requirements of allocated units.

[0038] To achieve adaptive scaling decisions, the system also introduces an intelligent resource analysis and scheduling module. Figure 4 As shown in the figure. This module uses machine learning algorithms to continuously analyze task operation data, learn the resource demand characteristics of different tasks at different stages, and predict future resource requirements based on this. Based on the prediction results, the scheduler combines factors such as task priority and data locality to match the optimal resource combination for each newly submitted task, thereby reducing scheduling overhead and improving task performance.

[0039] The core of the prediction model is a long short-term memory neural network (LSTM). For the scenario of the present invention, the forward propagation process of the model can be expressed by the following formula: in, is the input feature vector at time t, including task operation indicators such as CPU usage, memory usage, and IO load; is the output state of LSTM at time t, representing the prediction result of resource demand; is the internal memory state of LSTM; and are the weight matrix and bias vector to be learned respectively; and is the activation function.

[0040] By training on massive historical task operation data, the model can capture the temporal characteristics and hidden patterns of task resource requirements, providing accurate prediction reference for scheduling decisions.

[0041] Compared with traditional heuristic scheduling, the predictive scheduling method of the present invention can significantly reduce resource conflicts and task queuing, and improve the overall service quality and resource utilization efficiency of the system.

[0042] The serverless task running environment is another major innovation of this platform. Figure 5 As shown in the figure, users can directly submit data processing code and task configuration without having to manage the server, and the platform can automatically complete the entire process of resource matching, environment deployment, task scheduling, and operation monitoring. The system first parses the user request and extracts key metadata, such as data source address, resource requirements, task priority, etc. Based on this information and the output of the resource prediction model, the scheduler uses a dynamic programming algorithm to search for the optimal resource allocation solution. The goal is to minimize the resource supply cost while meeting the user's SLA. The optimization problem can be described as: in, Indicates whether to assign the task resource units, represents the cost of allocating the resource unit, Represents the resource requirements of the task at time t. Constraints ensure that the resource supply meets the actual demand at any time.

[0043] After completing resource allocation, the system will automatically deploy and configure the task running environment, including the operating system, middleware, and dependent libraries, and start the Serverless function to execute the actual computing logic. The life cycle of the function is fully controlled by the platform, which can dynamically expand and recycle computing units as needed, and continuously monitor resource usage and task progress, realizing the true "user submits code, and the platform takes care of everything."

[0044] In order to support performance isolation in a multi-tenant scenario, the present invention also designs a multi-tenant management mechanism based on resource quotas and priorities.

[0045] like Figure 6 As shown in the figure, tasks of different tenants are logically divided into independent resource pools and managed by a unified resource allocator. When allocating resources, first check the available quota of the corresponding tenant resource pool and queue them strictly according to the priority until the quota is exhausted. The resource allocation problem can be formalized as a multi-constrained knapsack problem: in, and Represented as tasks the benefits and costs of allocating resources, Indicates tenant By solving this optimization problem, the system can maximize the overall benefits in a multi-tenant scenario while meeting the quota limit.

[0046] Finally, the visual monitoring management module is as follows Figure 7 As shown in Figure 1, it provides administrators with a global view of system operation. By summarizing multi-dimensional performance indicators, the module can reflect the health status of the platform in real time and detect abnormal risks in a timely manner.

[0047] At the same time, the system has built-in multiple anomaly detection algorithms that can proactively identify performance bottlenecks and potential faults, and implement intelligent operation and maintenance. Anomaly detection is based on statistical modeling of data features. Taking CPU usage as an example, the Gaussian distribution assumption is used: By estimating the parameters of normal time series data, the normal range of CPU usage is obtained If multiple consecutive sampling points exceed this range, it is considered abnormal and an alarm is issued. Administrators can also take proactive countermeasures based on system warnings and combined with visual information, such as adjusting system configuration and optimizing task layout, to minimize abnormal escalation.

[0048] In summary, this embodiment has systematically improved and innovated the traditional big data platform from multiple levels, forming a complete Serverless distributed computing solution. Key designs such as elastic resource management, intelligent scheduling, multi-tenant isolation, and visual operation and maintenance help to greatly simplify the application development process, optimize resource allocation efficiency, and improve system robustness, providing strong support for the accelerated implementation of data-driven applications in the big data era.

[0049] Example 2: Based on Example 1, this example further optimizes the construction method of the elastic resource pool and introduces heterogeneous computing capabilities to adapt to diverse big data computing needs.

[0050] Traditional homogeneous clusters can hardly fully meet the stringent requirements of current big data applications in terms of storage and computing density, memory bandwidth, and energy efficiency.

[0051] For this reason, Figure 8 As shown, this embodiment adds multiple heterogeneous computing units such as GPU and FPGA on the basis of the original general x86 cluster to form a fusion heterogeneous cluster. Different computing units complement each other in terms of architecture, instruction set, parallelism, etc., and can give full play to their respective strengths, thereby improving overall benefits in multiple dimensions such as performance, cost, and energy efficiency.

[0052] Specifically, GPUs, with their highly parallel multi-core architecture and memory bandwidth advantages, have demonstrated outstanding performance in deep learning training, graph computing and other scenarios, and can serve as a powerful supplement to general-purpose CPUs. On the one hand, the platform supports users to directly submit jobs optimized for GPUs; on the other hand, for regular jobs, the system will intelligently analyze their computing characteristics, and when dense matrix operations and vectorized calculations are found, it will try to automatically offload them to the GPU for implicit acceleration. Compared with pure CPU solutions, GPU acceleration can shorten the training time of tasks such as machine learning by several to dozens of times.

[0053] FPGA combines the high efficiency of hardware with the flexibility of software. Through deeply customized computing pipelines, the server CPU can be freed from many data-intensive tasks, such as compression / decompression, encoding and decoding, encryption, string matching, data sorting, etc. These tasks with low computing efficiency on the CPU can achieve orders of magnitude performance improvement once they are offloaded to the FPGA, and the energy consumption is much lower than that of the GPU. The platform pre-installs FPGA implementations of common data processing operators, and supports users to upload FPGA bitstreams of custom operators. The system can automatically identify and call them according to the characteristics of the workload.

[0054] In order to give full play to the performance of heterogeneous hardware, a hardware perception module is added to the scheduler. This module is responsible for perceiving and comparing the computing power of different hardware and prioritizing the scheduling of tasks to the most suitable device for execution.

[0055] At the same time, secondary factors such as energy consumption and cost are weighed to improve the overall performance ratio while meeting the latency SLA. In addition, an affinity evaluation mechanism is introduced to schedule related tasks belonging to the same DAG to the same device as much as possible to reduce cross-device data transmission. For strongly interfering tasks that cannot be isolated, they will be scheduled to different devices to avoid contention.

[0056] The introduction of heterogeneous clusters has injected new vitality into the big data processing platform. On the one hand, different computing units can play to their strengths and avoid their weaknesses, complement each other and work together to maximize the utility of limited resources; on the other hand, targeted optimization of scheduling and programming frameworks for hardware differences can match data processing requirements at a finer granularity, allowing users to gain an experience that exceeds that of homogeneous clusters. This also reflects the new trend of deep integration of big data technology and hardware architecture.

[0057] In addition, this embodiment also focuses on optimizing the storage and network IO performance of the big data platform. Many big data analysis tasks today need to process data sets at the PB level, which poses a great challenge to data reading and writing efficiency.

[0058] To this end, the platform uses an all-flash distributed storage cluster, with a single node providing up to several GB / s of IO throughput and access latency controlled at hundreds of microseconds. Compared with mechanical hard disks, flash memory has obvious advantages in performance, energy consumption, reliability, etc., and as the cost per unit capacity continues to decline, it has gradually become economically feasible for large-scale deployment.

[0059] In addition, the storage system adopts a flat design, and the computing framework can directly access the underlying block devices, minimizing the software stack overhead. At the same time, storage and network are deeply integrated, supporting high-speed IO transmission protocols such as RDMA and NVMe-oF, and bringing the IO performance between computing and storage, and between storage nodes to a new level. Distributed storage also provides a flexible replica placement strategy, which can balance factors such as storage media and cross-rack backup according to the locality and reliability requirements of the task to achieve the optimal data layout.

[0060] In terms of network, the platform uses the SDN concept to reconstruct the data center network. 25 / 100GbE high-speed Ethernet is deployed in a single rack, and InfiniBand with higher bandwidth is used between racks. Built on the CLOS topology, the network can provide ultra-high internal aggregate bandwidth and extremely low hop count. Based on the Host-SM architecture, intelligent hierarchical scheduling is introduced, small traffic is accelerated by RoCE, large traffic is accelerated by RDMA fast channel, storage traffic and computing traffic are intelligently separated, and the overall network utilization and QoS are significantly improved.

[0061] The organic combination of high-performance storage and intelligent networks has comprehensively upgraded the efficiency of data supply. This is crucial for many IO-sensitive big data applications, especially in the current situation where data volume continues to expand, and can prevent IO from becoming a performance bottleneck for the system.

[0062] At the same time, unified job scheduling enables better coordination of computing, storage, and network resources to achieve end-to-end pipeline optimization. These innovations provide a solid foundation for the continuous optimization of data-intensive applications in terms of latency, throughput, and cost.

[0063] In summary, this embodiment has innovated the hardware infrastructure of the big data platform from multiple perspectives. On the one hand, the heterogeneous cluster integrating multiple paradigms has brought about an all-round improvement in computing power; on the other hand, the joint optimization of storage and network systems has enabled a leapfrog development in data supply performance. The synergy of software and hardware, and the integration of computing and storage, have jointly built a solid foundation for the new generation of big data processing platforms, which will help various industries cope with the continuous impact of the data torrent and more efficiently discover value from massive data.

[0064] Example 3: Based on the previous example, this example further optimizes the design of the Serverless task running environment and introduces a workflow scheduling mechanism to simplify the definition and orchestration of complex tasks.

[0065] Specifically, if Fig. 9 As shown in the figure, a lightweight workflow engine is integrated into the task manager. Users can use a declarative domain-specific language (DSL) to describe the dependencies between tasks, such as execution order, data transfer, conditional branches, exception handling, etc., to form a directed acyclic graph (DAG) with tasks as nodes and dependencies as edges. Each task node can be regarded as a Serverless function, which contains meta-information such as code, configuration, and resource requirements. Compared with the imperative task submission method, this declarative task definition is more concise and intuitive, fully hiding the underlying scheduling details, allowing users to focus on the business logic itself.

[0066] To achieve efficient execution, the workflow engine will hierarchically split and optimize the DAG to form multiple non-intersecting task subsets, each of which corresponds to a minimum schedulable unit. When allocating resources, the scheduler will comprehensively consider the common characteristics of tasks in the subset, such as shared access to the same data set, dependence on the same operating environment, etc., and deploy them in the same computing unit as much as possible to reduce unnecessary data movement and task startup overhead. For task chains with synchronous dependencies, they are scheduled to the same process first, and data is transmitted through efficient in-process pipelines. For asynchronous dependencies, when the upstream task ends, the intermediate result data is persisted to distributed storage, and downstream tasks can obtain it on demand.

[0067] It is worth mentioning that the workflow engine will adaptively and dynamically adjust the task DAG. On the one hand, it determines the selection of subsequent branches based on the intermediate results of the task operation; on the other hand, it dynamically adjusts execution parameters such as concurrency and cache strategy based on real-time feedback on task operation efficiency. This adaptive optimization ensures that the execution efficiency of the workflow can be continuously improved with changes in data characteristics and cluster status, avoiding the limitations of static scheduling solutions.

[0068] After the introduction of the workflow mechanism, the Serverless computing model has been extended from a single function to the entire application. This not only greatly reduces the difficulty of orchestrating complex tasks, but also enables the performance advantages of distributed computing to be fully reflected in higher-level application logic. Users can easily deploy and expand massive concurrent Serverless workflows without having to understand and intervene in the allocation and recycling of underlying resources, truly realizing a new paradigm of "Big Data as a Service".

[0069] In summary, the present invention has systematically innovated the architecture and implementation of the traditional big data processing platform from multiple perspectives, and proposed a new distributed computing framework based on the Serverless concept. The core features of the invention include: (1) Designed for big data scenarios and matching the resource requirements of data-intensive applications; (2) Introducing the Serverless paradigm, users no longer need to manage servers, lowering the usage threshold; (3) A two-level resource model is adopted, combined with intelligent scheduling, to achieve fine-grained resource sharing and performance isolation; (4) Provide on-demand, automatic elastic scaling capabilities to flexibly respond to load fluctuations; (5) Utilize heterogeneous hardware collaboration and a new storage architecture to break through IO bottlenecks and release computing power;

[0070] The present invention has broad application prospects in many fields such as industrial production, telecommunications operations, and financial risk control. For example, in the industrial Internet scenario, this platform can be used to collect, process, model and analyze data throughout the equipment life cycle, and assist in intelligent predictive maintenance; in the telecommunications industry, real-time correlation calculations can be performed on multi-source network signaling data to achieve precise network optimization and fault location; in the financial field, multi-dimensional heterogeneous data can be integrated to carry out large-scale real-time risk measurement and support millisecond-level business decision-making. At the same time, the innovative architecture and key technologies proposed by the present invention are also of great significance for promoting the integrated development of big data, cloud computing, artificial intelligence and other fields.

[0071] It should be noted that the above embodiments are only preferred implementations of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art will appreciate that, without departing from the spirit and substance of the present invention, various equivalent transformations and modifications to these embodiments, such as adapting the system architecture and configuration parameters according to the application scenario requirements, or introducing other optimization algorithms and strategies, should all be regarded as the scope of protection of the present invention. In short, the scope of protection of the present invention should be defined by the claims.

Claims

1. A distributed computing platform based on Serverless architecture for big data computing scenarios, characterized by: include: A distributed elastic computing resource pool consisting of multiple computing nodes, wherein the computing nodes are divided into multiple lightweight computing units; A dual scaling mechanism for horizontally scaling the computing resource pool and vertically scaling a single computing unit; An intelligent resource analysis and scheduling module based on machine learning, which is used to analyze the resource consumption patterns of computing tasks and predict future demand, and optimize the scheduling strategy accordingly; Supports the Serverless task running model that makes big data tasks serverless. After the user submits the code, the system automatically schedules resources and charges on demand. Perform unified abstract management of computing resources and provide multi-tenant resource isolation and performance guarantee mechanisms; A visual monitoring platform that displays core indicators and provides intelligent optimization solutions based on big data analysis.

2. The distributed computing platform according to claim 1, characterized in that: The elastic computing resource pool supports dynamic horizontal scaling according to overall load demand to provide sufficient computing resource supply.

3. The distributed computing platform according to claim 1, characterized in that: The lightweight computing units divided by the computing nodes are containers to support fine-grained resource scheduling.

4. The distributed computing platform according to claim 1, wherein: The dual scaling mechanism firstly copes with large load fluctuations by horizontally scaling the computing resource pool, and secondly implements real-time resource adjustment of started tasks by vertically scaling a single computing unit.

5. The distributed computing platform according to claim 1, wherein: The intelligent resource analysis and scheduling module adopts a machine learning algorithm to establish a resource consumption prediction model through training and learning of historical data and real-time monitoring data, and adjusts the resource configuration of hot task tasks in advance accordingly.

6. The distributed computing platform according to claim 1, wherein: The Serverless task running model decouples task code from underlying resources, provides automatic resource provisioning and on-demand billing, and relieves users of the operation and maintenance burden.

7. The distributed computing platform according to claim 1, wherein: The multi-tenant resource isolation mechanism strictly isolates resource usage of different tenants and provides performance guarantees for key tasks.

8. The distributed computing platform according to claim 1, wherein: The visual monitoring platform intuitively displays core operating indicators such as resource pool status and task distribution through charts and other means, and provides intelligent diagnosis and optimization solutions based on big data analysis.

9. The distributed computing platform according to any one of claims 1 to 8, characterized in that: The platform further includes a scheduling policy configuration module, and the administrator can flexibly intervene in scheduling behaviors such as resource allocation and task priority through policy configuration.

10. The distributed computing platform according to any one of claims 1 to 8, characterized in that: The machine learning algorithms include but are not limited to decision trees, support vector machines, neural networks, etc.

Images