Big data task scheduling method based on cloud native technology

Abstract

This invention discloses a big data task scheduling method based on cloud-native technology, comprising the following steps: (a) establishing a big data computing task model; (b) establishing a big data service scheduling sequence model: when container cluster r works on task k in the processing queue corresponding to a subset, the processing result is sent to a communication queue indexed by (k, r), where the rate is μ. (k，j) When container cluster r broadcasts the result of processing task k in the communication queue indexed by (k, r), the processing result is sent to the corresponding container cluster with rate b. j / c k (c) Propose a maximum weight type scheduling strategy: at time n, a vector z is assigned based on the maximum weight strategy given the sequence length. Through the combination of the above steps, the load on major data computing nodes in a distributed environment can be dynamically balanced, computing resources can be effectively utilized, big data collaborative computing tasks can be completed quickly, and the complexity of scheduling strategies for big data service scheduling problems can be reduced.

Description

Technical Field

[0001] This invention belongs to the field of big data platform technology and relates to a task scheduling method, specifically a big data task scheduling method based on cloud-native technology. Background Technology

[0002] A big data platform (i.e., a big data infrastructure platform) is a common service support environment for big data applications. It provides common services such as operation support, development integration, and user permissions for big data applications. At the same time, it provides basic services such as resources and operation and maintenance for big data collection platforms, data governance platforms, asset management platforms, and data mining platforms, supporting the construction of a common foundation and ecosystem for the operation of the big data system.

[0003] The big data infrastructure platform is the foundation of the big data application system, providing a basic environment and functional component support for big data analysis, processing, and business applications. It not only undertakes basic big data management but also provides data support for upper-layer applications. Simultaneously, based on various big data computing engines, it supports big data mining and analysis. Addressing the distributed storage, computing, analysis, mining, modeling, and business query needs of various business systems for massive amounts of big data, the big data platform supports complete data processing covering the entire data lifecycle by providing basic resource and data services.

[0004] like Figure 1 As shown, existing big data platforms employ traditional software architectures, resulting in large software volumes, complex migration and scheduling, and primarily operating in cloud centers. Traditional big data architectures require large-scale and highly dynamic computing and storage resources; therefore, resource optimization and scheduling capabilities directly impact the reliability and efficiency of big data operations. While big data platforms themselves provide strong resource scheduling capabilities, increasingly demanding resource scheduling requirements, particularly for cloud-edge collaboration and edge application scenarios, necessitate more specialized resource scheduling capabilities to meet the demands for highly reliable and available scheduling of limited resources.

[0005] Based on the existing big data platform architecture, the platform mainly includes the following capabilities: First, it needs to manage a full-featured computing engine (infrastructure), providing a unified large-scale computing environment covering offline computing, real-time stream computing, and graph computing, capable of utilizing the elastic computing resources provided by the infrastructure to achieve parallel processing of large-scale data. Simultaneously, it needs to have open technical interfaces, compatible with mainstream data development standards and frameworks, and support the processing of various heterogeneous data; it needs to support computing engines based on architectures such as Hadoop, Spark, Flink, or domestically, MaxCompute and Blink. The computing engine should be built on cloud computing infrastructure, including servers, storage, networks, virtualization, etc., which is the "foundation" of the big data platform. Through the cloud platform, pooled management of computer resources such as network, computing, and storage is achieved, providing the computing and storage cloud platform required for large-scale computing operations. Simultaneously, combined with a dynamically scalable, highly available architecture, resources are elastically allocated according to demand, ensuring the security and isolation of resource usage. Traditional big data platforms mainly provide a unified resource scheduling and operation and maintenance management platform. Through a unified data asset catalog, they support multiple cloud nodes to jointly complete complex cross-domain collaborative computing, thereby meeting the basic service needs of computing and storing massive amounts of data continuously generated by multiple units and different specialties within an organization. Secondly, it supports full-stack productivity tools, with a focus on ensuring data development, governance, and service capabilities. This requires building unified data resource assurance, data governance scheduling, and common data application services. From a business perspective, data resource assurance focuses on how data is aggregated, integrated, and used for data mining and analysis; data governance scheduling focuses on data quality and evaluation; and data application focuses on data service and usage. Through a big data platform, a unified data access mode is provided, adapting to various data sources and acquiring data in various formats. It also provides multi-dimensional data cleaning, processing, modeling, and storage, achieving unified data resource aggregation and integration. Based on batch and stream processing frameworks, methods such as data redundancy removal, anomaly detection, and normalization are used to clean raw data, achieving data preprocessing. Different data management engines, such as distributed file systems, NoSQL databases, relational databases, and time-series databases, are used to achieve partitioning, storage, cataloging, and indexing of massive aerospace data. Mathematical statistics and machine learning algorithms are used to achieve clustering, association, and predictive analysis of historical, real-time, and time-series data. Various algorithm models are built using expertise in mechanical, electronic, and physical fields combined with aerospace expertise. Finally, by combining middleware technologies such as microservices, we provide data applications with data management and analysis capabilities, reusable microservice components that solidify experiential knowledge, and a corresponding one-stop development environment, including the underlying technology platform and the development tool platform for end developers.

[0006] Existing big data platforms primarily focus on data collection, data component installation, and data warehouse construction. Their development, management, and operation are scattered across various big data components, lacking a unified management system. Furthermore, each component in a traditional big data platform has its own installation and management process, increasing the difficulty of platform construction and operational complexity. Secondly, big data platforms and cloud computing platforms are typically built in parallel and deployed independently, resulting in high resource consumption and significant duplication of effort. Finally, and more importantly, with the development of edge computing and IoT technologies, big data applications need to return to their origins, moving closer to the source of big data generation—the edge and endpoint—to achieve real-time big data processing and application locally. However, existing big data infrastructure platforms are large in scale and difficult to deploy, making it difficult to meet the needs of cloud-edge collaboration and lightweight edge applications.

[0007] In recent years, with the rapid development of cloud-native technology, which involves building and running application services directly in the cloud (as opposed to traditional data centers), various systems' business or application services can thrive on the cloud. Gartner predicts that over 95% of information systems may adopt cloud-native platforms in the coming years. Therefore, cloud-native platforms hold immense potential in the next few years. Building big data platforms based on cloud-native technology ensures that cloud computing and big data are hosted on a unified foundation. Especially with the rise of container technology, numerous excellent software systems have emerged around it. Google began internally implementing container technology as early as 2006, highlighting its low overhead and fast startup speed. Later, Docker innovatively proposed container image packaging technology, resolving software runtime environment dependency conflicts during software distribution. This allows developers to compile applications once and deploy them multiple times to multiple environments. Dockerfiles define all container runtime environments and startup behaviors, strictly limiting service runtime environments and significantly reducing operational complexity caused by runtime environments and startup methods during software distribution. Kubernetes, based on years of experience with Google Borg, further provides cluster business abstractions such as resource scheduling, service orchestration, and service discovery, solving the problem of large-scale container service deployment. By leveraging cloud-native technologies to organically integrate cloud computing platforms and big data system components, achieving containerization and service-orientation, we can create efficient business big data processing and application capabilities. This allows for a more comprehensive and timely understanding of business conditions across various industries, accelerating the iteration speed of big data-driven application services. Furthermore, it enables unified management of system assets across different systems, establishing a service-oriented operation system and enhancing data-driven service transformation. When using cloud-native technologies, it is necessary to reduce the complexity of big data task scheduling to ensure its effectiveness and improve resource utilization. Summary of the Invention

[0008] To address the aforementioned shortcomings, this invention provides a big data task scheduling method based on cloud-native technology, thereby reducing the complexity of scheduling strategies in big data service scheduling problems.

[0009] To achieve the above objectives, the present invention provides a big data task scheduling method based on cloud-native technology, comprising the following steps:

[0010] (a) Establish a big data computing task model: Model each business of big data task scheduling as a big data service graph, and define... This corresponds to the big data service graph representation of type m, where Represents a node set of type m, ε m c represents the edge set of big data services. k This indicates a task whose output size is of type k.

[0011] (b) Establish a big data service scheduling sequence model: when container cluster r is in the corresponding subset When working on task k in the processing queue, the processing result is sent to... A communication queue indexed by (k, r), where the rate is μ (k，j) When container cluster r is in the process of being... When the result of processing task k is broadcast in the communication queue with index (k, r), the processing result is sent to the queue corresponding to the rate b. j / c k subset of The processing queue;

[0012] (c) Propose a maximum weight type scheduling strategy: at time n, given the sequence length The maximum weight strategy assigns a vector z:

[0013]

[0014]

[0015] in, It is a vector representing the queue length at time n.

[0016] Optimally, in step (a), when k and k′ belong to the same big data service, and (k′, k)∈ε m Then task k' is the parent task of task k; let Let k be the parent set of k. When k and k′ belong to the same big data service, and there exists a directed path from k′ to k in that big data service, then task k is said to be a descendant of task k′.

[0017] Ideally, in step (b), the processed task outputs are sent to the same container cluster.

[0018] Furthermore, in step (b), when there are M big data service graphs, a sequence is maintained if and only if for all All of k's descendants are also In; and by maintaining a by A sequence of (k, r) indices, used for stages The processable task k in the middle.

[0019] Furthermore, in step (b), the action for processing task k is defined as... The corresponding processing activities in the processing queue, and defined. and They are sets of different processing activities and different communication activities in the network, and

[0020] but

[0021] Where K' is the number of sequences in the network, and k′ (1≤k′≤K′) represents the index sequence; corresponding to activity a, the rate at which sequence k′ changes is d. k′a ;

[0022] When the vector e = [e k′ And since the first stage corresponding to sequence k′ has been processed, then e k′ =λ m ; It is an event The allocation vector.

[0023] Ideally, in step (c),

[0024] Define the fluid quantity in sequence k′ as The fluid dynamics are as follows: X(t) = X(0) + et + DT(t);

[0025] Where, X(t)=[X k′ [t] is a vector representing the queue length, T a (t) is the total time of the activity, T(t) = [T a [t] is a vector of the total service time for different activities, and V(t) = (X T Using X) / 2 as the Lyapunov function, the drift of V(t) can be calculated as follows:

[0026]

[0027] Among them, z * It is a feasible allocation vector; when e∈Λ″, then At this point, the fluid model is weakly stable, while the queuing network is rate-stable.

[0028] Furthermore, in step (c), when e is inside the capacity region Λ″, then

[0029] This invention provides a big data task scheduling method based on cloud-native technology. Through the combination of the above steps, it can dynamically balance the load on major data computing nodes in a distributed environment, effectively utilize computing resources, quickly complete big data collaborative computing tasks, and reduce the complexity of scheduling strategies for big data service scheduling problems. Attached Figure Description

[0030] Figure 1 A comparison diagram of traditional and cloud-native big data architectures;

[0031] Figure 2 This is a schematic diagram of the big data service used for task scheduling in this invention;

[0032] Figure 3 This is a schematic diagram of the task sequence network of the present invention;

[0033] Figure 4 This is a queuing network diagram with low complexity for this invention;

[0034] Figure 5 This is a diagram of the cloud-edge collaborative big data platform architecture of the present invention;

[0035] Figure 6 This is a schematic diagram of the big data task allocation process of the present invention;

[0036] Figure 7 This invention provides a comparison between container-based service deployment and traditional methods.

[0037] Figure 8 This is a business process diagram of the big data support system of the present invention;

[0038] Figure 9 This is a data flow diagram of the big data support system of the present invention. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art under the premise of equivalent changes and modifications should fall within the protection scope of the present invention.

[0040] This invention relates to a big data task scheduling method based on cloud-native technology, comprising the following steps:

[0041] (a) Establish a big data computing task model: Model each business of big data task scheduling as a big data service graph (e.g., Figure 1 As shown), definition This corresponds to the big data service graph representation of type m, where Represents a node set of type m, ε m c represents the edge set of big data services. k This indicates a task whose output size is of type k.

[0042] When k and k′ belong to the same big data service, and (k′, k)∈ε m Then task k' is the parent task of task k; let Let k be the parent set of k. When k and k′ belong to the same big data service, and there exists a directed path from k′ to k in that big data service, then task k is said to be a descendant of task k′.

[0043] (b) Establish a big data service scheduling sequence model: When designing the big data service scheduling sequence model, ensure that the output of processed tasks is sent to the same container cluster. When there are M big data service graphs, (1) for By maintaining a sequence if and only if for all All of k's descendants are also (2) For container cluster r, by maintaining a [structure / structure]... A sequence of (k, r) indices, used for stages The processable task k in the context. In this embodiment, the business specified by the big data service and the network consisting of two container clusters, the corresponding sequence network is as follows: Figure 2 As shown.

[0044] When container cluster r corresponds to the subset When working on task k in the processing queue, the processing result is sent to... A communication queue indexed by (k, r), where the rate is μ (k，j) When container cluster r is in the process of being... When the result of processing task k is broadcast in the communication queue with index (k, r), the processing result is sent to the queue corresponding to the rate b. j / c k subset of The processing queue;

[0045] Define the action for processing task k as... The corresponding processing activities in the processing queue, and defined. and They are sets of different processing activities and different communication activities in the network, and

[0046] but

[0047] Where K' is the number of sequences in the network, and k′ (1≤k′≤K′) represents the index sequence; corresponding to activity a, the rate at which sequence k′ changes is d. k′a When the vector e = [e] is reached k′ And since the first stage corresponding to sequence k′ has been processed, then e k′ =λ m Otherwise e k′ =0; It is an event The allocation vector.

[0048] (c) Propose a maximum weight type scheduling strategy: at time n, given the sequence length The maximum weight strategy assigns a vector z:

[0049]

[0050]

[0051] in, It is a vector representing the queue length at time n.

[0052] Define the fluid quantity in sequence k′ as The fluid dynamics are as follows: X(t)=X(0)+et+DT(t) (5);

[0053] Where, X(t)=[X k′ [t] is a vector representing the queue length, T a (t) is the total time of the activity, T(t) = [T a [t] is a vector of the total service time for different activities, and V(t) = (X T Using X) / 2 as the Lyapunov function, the drift of V(t) can be calculated as follows:

[0054]

[0055] Among them, z * It is a feasible allocation vector; when e∈Λ″, then At this point, the fluid model is weakly stable, and the queuing network is velocity stable. When e is inside the capacity region Λ″, then... After optimization, the four nodes with the queuing network are as follows: Figure 3As shown, the queuing network for the business phase sequence has K queues, which greatly reduces the complexity of the scheduling strategy for the big data service scheduling problem. That is, through container-level computing and network services, the load on major data computing nodes in a distributed environment can be dynamically balanced, effectively utilizing computing resources and quickly completing big data collaborative computing tasks. In other words, through the combination of the above steps, the load on major data computing nodes in a distributed environment can be dynamically balanced, effectively utilizing computing resources and quickly completing big data collaborative computing tasks, thus reducing the complexity of the scheduling strategy for the big data service scheduling problem.

[0056] With the continuous development of cloud-edge collaboration platforms, the field of big data processing is also quietly changing. Although the main source of data is still user-based, it is gradually shifting towards "things". With richer data sources, multiple factors, including network communication technology and data processing methods, are jointly driving the data processing model to gradually shift from stand-alone and cloud models to edge and cloud-edge collaboration models.

[0057] The aforementioned big data task scheduling method based on cloud-native technologies can be applied to cloud-edge collaborative big data platform architectures. For example... Figure 5 As shown, this cloud-native architecture-based cloud-edge collaborative big data platform architecture can support efficient big data services in the cloud-edge collaborative mode, while solving problems such as unified scheduling and management of heterogeneous computing resources.

[0058] To ensure the effectiveness of the above task scheduling and improve resource utilization, it is usually necessary to decompose larger tasks before execution. The task decomposition process consists of three main stages: (1) the task issuing node sends a computation request to the network; (2) each available node in the network feeds back its node availability and computational resource status to the task issuing node level by level; (3) the task issuing node decomposes the task according to the sum of the computational resources of the nodes directly connected to it and its subsequent nodes in the network. After the total computational task is initially decomposed to each connected node, it is then decomposed a second time and finally allocated to all available nodes. Figure 6The diagram illustrates the task allocation process. After node A generates a computational task, it cannot complete the entire computation on its own. Therefore, it requests other computing nodes in the network with task computation capabilities to perform collaborative computation. Nodes B and C communicate directly with A. The computational resource information of nodes D and E is transmitted to node A via B. After receiving information from all network nodes, node A decomposes the task based on the set of nodes B, D, and E, along with node C. The decomposed subtasks are then passed to nodes B and C. Node B further decomposes the task and forwards it to nodes D and E. Based on the allocation, the task is distributed to the distributed computing network using various distributed computing messages for the next step of the task scheduling process. In the task decomposition process, in addition to considering the choice of task allocation method, the task granularity also needs to be considered. The overall size and number of a computational task after being decomposed into multiple subtasks can be defined as the task granularity. An effective task decomposition strategy and task granularity design are prerequisites for computational task decomposition planning. The decomposition granularity of distributed computing tasks has a significant impact on the management of a computational task. Therefore, it is necessary to select an appropriate granularity for computational task decomposition based on the specific scenario.

[0059] To achieve rapid cloud-edge deployment of big data services, the first challenge is the rapid deployment of individual services. Utilizing container image technology transforms the service deployment model from the traditional layer-by-layer installation approach to a "copy-and-run" model, saving online installation and configuration time. Combined with the rapid startup capabilities of containers, this enables the rapid deployment of individual services. Figure 7This paper compares the differences between service deployment based on container images and traditional installation and deployment processes. Container image technology is a software packaging technology that packages the executable program of a service with the software environment that the service depends on at runtime into an image file. The installation of dependent software and the configuration of internal parameters are completed by developers during the development phase. During the service deployment and runtime phase, the container management environment only needs to allocate a container that meets the resource and access address requirements of the service according to the service description file, load the container image into memory, and start it, making the service available immediately. Compared with the traditional software deployment process of installing and configuring the operating system, database, dependent software, and application software layer by layer on-site, it saves a lot of installation and configuration time. At the same time, container image files can be easily copied to any node with a pre-installed container management environment for rapid deployment, avoiding the problem of repeated installation and configuration on each node in the traditional installation and deployment process. This demonstrates greater deployment efficiency when deploying on a large number of nodes simultaneously. Containers, as a lightweight virtualization technology, isolate various resources of the host machine into independent containers on demand. The resources of each container are independent and do not affect each other. Therefore, each service runs in its own independent resource space, avoiding dependency environment and configuration conflicts between multiple services. This solves the problem of installation and deployment failures often caused by software conflicts in traditional deployment processes, increasing the success rate of deployment. Unlike traditional virtualization technologies, container virtualization shares the host machine's operating system kernel, eliminating the need to install a virtual operating system for each container. Therefore, container-based big data services start much faster than traditional services, achieving second-level startup speeds. At the same time, the lightweight virtualization capabilities of containers significantly reduce the resource consumption of big data services, facilitating the promotion and application of big data services to the edge and end devices. In short, containers, as a lightweight virtualization technology, isolate various resources of the host machine into independent containers on demand. The resources of each container are independent and do not affect each other. Therefore, each service runs in its own independent resource space, avoiding dependency environment and configuration conflicts between multiple services. This solves the problem of installation and deployment failures often caused by software conflicts in traditional deployment processes, reducing the difficulty of big data deployment and improving operational efficiency.

[0060] The main business process design and data process design of the cloud-native architecture-edge collaborative big data platform architecture described above are as follows: Figure 8 and Figure 9 As shown.

[0061] The main business process design is as follows: (1) The business process of the big data support system based on cloud-native architecture is service-oriented. Through service-oriented data introduction, aggregation, cleaning, fusion, analysis, and reconstruction processes, it provides work support, business support, and data support services for data collection, data analysis, data operation and maintenance, and data application. (2) The data integration service introduces various data contents from the data source into the big data support platform through data introduction. Through data collection, data import, data loading, extraction and transformation, data encapsulation, data transmission, cluster management, and configuration management, data production personnel introduce and collect source data, and store the data in the big data storage service and provide it directly to the data sharing service through the data service interface. (3) The data collected by the data integration service will be stored in the big data storage system. The big data storage service performs layered storage, storage management, model management, scheduling management, operation management, and data asset management of data according to files, relational databases, and non-relational databases, and provides relevant data support for the reconstruction service of data tables. Big data computing provides the big data infrastructure platform with functions such as real-time computing, ad-hoc computing, offline computing, in-memory computing, streaming computing, and parallel batch computing, and provides scheduling services for computing resources and algorithms through resource management and task scheduling services. Data governance provides the big data infrastructure platform with data maintenance, data governance standards, data imprinting, quality assessment, and version management services, and manages the data governance model and governance level. (4) Data analysis service uses the data resources provided by the data table reconstruction service to provide data support for data analysts. Data analysts can perform data analysis in the form of statistical analysis, mining analysis, visual analysis, and report analysis on the reconstructed data. The system can provide data and analysis services to other business systems, and provide support services for reconstructed data and multidimensional analysis data to other business systems. Data sharing service provides data sharing services to the data processing center and edge processing nodes by accessing the source data of the data integration service and the business data reconstructed by the data table reconstruction service.

[0062] The data flow is designed as follows: (1) After collecting the data from the data source, the data integration system of the cloud-native big data support platform performs extraction, transformation, and encapsulation operations on the data, and transmits the full amount of data to the big data storage service for data storage, and provides the source data for data sharing service according to the data sharing service requirements. (2) The data stored in the big data storage service is provided to the data table reconstruction system for data table reconstruction service. (3) The data reconstruction service uses the computing and processing resources of the data computing service and data governance service to reconstruct the data according to the business reconstruction requirements; the reconstructed data is distributed as follows: (4) It is transmitted to the data analysis service for further data mining, and the final data will serve the data mining needs and support other business needs; it is transmitted to other business systems to provide intelligent data support services for their unit applications; it is transmitted to the data sharing service, which distributes the data to the data processing center and edge nodes.

[0063] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. At the same time, those skilled in the art should understand and implement the above description. Therefore, any equivalent changes or modifications made without departing from the concept disclosed in the present invention should be covered within the scope of protection of the present invention.

Claims

1. A big data task scheduling method based on cloud-native technology, characterized in that, Includes the following steps: (a) Establish a big data computing task model: Model each business of big data task scheduling as a big data service graph, and define... This corresponds to the big data service graph representation of type m, where This represents a set of nodes of type m. Represents the edge set of big data services. Indicates the output task k; (b) Establish a big data service scheduling sequence model: when container cluster r is in the corresponding task subset When working on task k in the processing queue, the processing result is sent to... The communication queue is indexed by (k, r), where the container cluster r retrieves task k from the processing queue at a rate of [value missing]. When container cluster r is in the process of being... When the result of processing task k is broadcast in the communication queue with index (k, r), the processing result is sent to the corresponding queue with rate subset of The processing queue; (c) Propose a maximum weight type scheduling strategy: at time n, given the sequence length The maximum weight strategy assigns a vector z: ； ； in, It is a vector representing the queue length at time n; In step (c), the sequence is defined. The fluid volume in is The fluid dynamics are as follows: ; in, It is a vector of queue lengths. It is a vector of the total service time for different activities. It is the a-th component of vector T(t), and ,Will As a Lyapunov function, it can be calculated to obtain The drift is: ; in, It is a feasible assignment vector; when ,but At this point, the fluid model is weakly stable, and the queuing network is rate-stable.

2. The big data task scheduling method based on cloud-native technology according to claim 1, characterized in that: In step (a), when k and k' belong to the same big data service, and If k and k' belong to the same big data service, and there exists a directed path from k to k' in that big data service, then task k is called a descendant of task k'.

3. The big data task scheduling method based on cloud-native technology according to claim 1, characterized in that: In step (b), ensure that the processed task outputs are sent to the same container cluster.

4. The big data task scheduling method based on cloud-native technology according to claim 3, characterized in that: In step (b), when there are M big data service graphs, a sequence is maintained if and only if for all k All of k's descendants are also In; and by maintaining a by A sequence of (k, r) indices for subsets The processable task k in the middle.

Images