Data processing method and device, computer storage medium and electronic device

Abstract

The present application discloses a data processing method and device based on a containerized application cluster management platform, an application scheduling method and device based on a containerized application cluster management platform, a computer storage medium and an electronic device, wherein the data processing method includes: obtaining data set information created based on a containerized application cluster management platform; storing and arranging the data set according to the occupied space of the data set in the data set information and the storage space of the computing node in the containerized application cluster management platform, and obtaining a set of schedulable computing nodes capable of storing the data set; preloading the data in the data set into the set of schedulable computing nodes, and onto the target computing node of the storage space that matches the occupied space of the data. This avoids the delay problem caused by the need for the application service to first request access to data from the storage node when executing, and improves the efficiency of subsequent data analysis and processing.

Description

Technical Field

[0001] The present application relates to the field of computer application technology, and more particularly to a data processing method and device based on a containerized application cluster management platform. The present application also relates to an application scheduling method and device based on a containerized application cluster management platform, a computer storage medium, and an electronic device. Background Art

[0002] Software development requires environment configuration, and the user's computer environment is very different. In order to ensure that the application software can run normally in different computer environments, and no longer need to provide suitable configuration files for different computer environments during software development, a solution of installing a virtual machine with an environment came into being. A virtual machine can run another operating system in one operating system, such as running a Linux system in a Windows system; thus, it can adapt to different computer environments. However, virtual machine technology has defects such as high resource usage, many redundant steps, and slow startup.

[0003] Based on the defects of virtual machine technology, container technology has gradually become the de facto standard for software delivery and operation. Container technology packages applications and application dependencies into container clusters, thereby eliminating the dependence of application operation on the underlying data center infrastructure. Since container technology can isolate processes, it can solve various defects in virtual machine technology.

[0004] Container technology requires container management tools to manage containers in order to change the way applications are deployed in large-scale data centers and implement service functions such as automated deployment, automatic expansion and reduction of container clusters.

[0005] Although container management tools perform well in stateless online applications and databases (MySQL), their shortcomings are more prominent in data-driven application environments such as online data analysis and machine learning. The main reason is that both online data analysis and machine learning are essentially using data to solve computer cognitive problems. The efficiency of data processing depends on the efficiency of data access. In the existing technology, in terms of data analysis and processing, considering data cost and management factors, it is necessary to separate data calculation and data storage in order to reduce costs and simplify management. However, the separation of data calculation and data storage will inevitably lead to problems such as data access delay, which in turn affects the efficiency of data processing and analysis. Summary of the invention

[0006] The present application provides a data processing method based on a containerized application cluster management platform to solve the problem of low data access efficiency in the prior art.

[0007] The present application provides a data processing method based on a containerized application cluster management platform, including:

[0008] Obtain the dataset information created based on the containerized application cluster management platform;

[0009] According to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set;

[0010] The data in the data set is preloaded into the set of schedulable computing nodes, on a target computing node of the storage space that matches the occupied space of the data.

[0011] In some embodiments, the obtaining of data set information created based on a containerized application cluster management platform is characterized by:

[0012] According to the application orchestration configuration file defined based on the containerized application cluster management platform, the created data set information is obtained.

[0013] In some embodiments, the step of performing storage arrangement on the data set according to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform to obtain a set of schedulable computing nodes capable of storing the data set includes:

[0014] According to the storage end specified in the data set information, query the occupied space of the data set;

[0015] Selecting candidate computing nodes that meet the screening requirements in the containerized application cluster management platform;

[0016] According to the occupied space information of the data set and the storage space of the candidate computing node, the data set is stored and arranged to obtain a set of schedulable computing nodes that can store the data set.

[0017] In some embodiments, selecting a candidate computing node that meets the screening requirements in the containerized application cluster management platform includes at least one of the following selection methods:

[0018] In the containerized application cluster management platform, selecting the candidate computing node that matches the label information defined in the data set information according to the label information of the computing node;

[0019] Selecting the candidate computing node that meets the availability zone range requirement defined in the data set information in the containerized application cluster management platform;

[0020] The candidate computing node that meets the specified device type defined in the data set information is selected in the containerized application cluster management platform.

[0021] In some embodiments, it also includes:

[0022] According to the space occupied by the data set or the usage requirement of the data set, the data set is segmented to obtain at least two data set blocks;

[0023] Recording the name of the data set block and the amount of the data set block corresponding to the name that needs to be allocated to the containerized application cluster into the created data set record object;

[0024] The step of performing storage arrangement on the data set according to the occupied space information of the data set and the storage space of the candidate computing node to obtain a set of schedulable computing nodes capable of storing the data set includes:

[0025] According to the occupied space of the data set blocks in the data set record object and the storage space of the candidate computing nodes, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set.

[0026] In some embodiments, it also includes:

[0027] Creating a computing node list for recording candidate computing node information according to the selected candidate computing nodes;

[0028] The step of performing storage arrangement on the data set according to the space occupied by the data set in the data set record object and the storage space of the candidate computing node to obtain a set of schedulable computing nodes capable of storing the data set includes:

[0029] According to the space occupied by the data set in the data set record object and the storage space of the candidate computing node in the computing node list, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set.

[0030] In some embodiments, the creating a computing node list for recording candidate computing node information according to the selected candidate computing node includes:

[0031] Sorting the candidate computing nodes in ascending or descending order according to the storage capacity information of the available storage space;

[0032] The computing node list is created according to the sorted candidate computing nodes.

[0033] In some embodiments, the step of performing storage arrangement on the data set according to the space occupied by the data set in the data set record object and the storage space of the candidate computing node to obtain a set of schedulable computing nodes capable of storing the data set includes:

[0034] Determine whether the computing node list has a set of available candidate computing nodes that meets a set availability threshold requirement;

[0035] If so, determining whether the space occupied by the data set is less than the storage capacity of the storage space of the available candidate computing nodes in the set of available candidate computing nodes;

[0036] If so, selecting an available candidate computing node that matches the occupancy from the set of available candidate computing nodes;

[0037] The available candidate computing nodes are determined as the set of schedulable computing nodes that can store the data set.

[0038] In some embodiments, it also includes:

[0039] When the determination of whether the occupied amount of the space occupied by the data set is less than the storage amount of the storage space of the available candidate computing nodes in the set of available candidate computing nodes is no, then determining whether the occupied amount of the space occupied by the data set is greater than the storage amount of the storage space of the available candidate computing nodes in the set of available candidate computing nodes;

[0040] If yes, selecting the available candidate computing node with the largest storage capacity from the range of available candidate computing nodes;

[0041] The amount of space occupied by the data set is updated, and the process returns to the step of determining whether the computing node list has a set of available candidate computing nodes that meets the set availability threshold requirement;

[0042] The step of determining the available candidate computing nodes as the set of schedulable computing nodes capable of storing the data set comprises:

[0043] The selected available candidate computing node with the largest storage capacity is determined as the set of schedulable computing nodes that can store the data set.

[0044] In some embodiments, it also includes:

[0045] When determining whether the occupied space of the data set is greater than the storage capacity of the storage space of the available candidate computing nodes in the set of available candidate computing nodes, if the result is no, selecting an available candidate computing node with the storage capacity equal to the occupied space from the range of the available candidate computing nodes;

[0046] The step of determining the available candidate computing nodes as the set of schedulable computing nodes capable of storing the data set comprises:

[0047] The selected available candidate computing nodes whose storage capacity is equal to the occupied capacity are determined as the schedulable computing node set.

[0048] In some embodiments, determining the available candidate computing nodes as the set of schedulable computing nodes capable of storing the data set includes:

[0049] The available candidate computing nodes are recorded in a scheduling list, where the scheduling list is used to record information about the set of schedulable computing nodes that can store the data set.

[0050] In some embodiments, it also includes:

[0051] When the determination of whether there is a set of available candidate computing nodes that meet the set availability threshold requirements in the computing node list is no, exit the step of performing storage arrangement for the data set according to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform to obtain a set of schedulable computing nodes that can store the data set, or re-execute the step of selecting candidate computing nodes that meet the screening requirements in the containerized application cluster management platform, or wait according to the set waiting time.

[0052] In some embodiments, it also includes:

[0053] When performing storage arrangement for the data set according to the space occupied by the data set in the data set record object and the storage space of the candidate computing node, determining whether the space occupied by the data set on the storage terminal changes;

[0054] If so, the data record object for recording the space occupied by the data set is recreated according to the data set after the storage amount is changed.

[0055] In some embodiments, it also includes:

[0056] Acquire at least one of storage capacity information of the data set in the set of schedulable computing nodes and access capacity information of data in the data set stored in the set of schedulable computing nodes;

[0057] The information is used as reference information for application scheduling in the containerized application cluster management platform.

[0058] The present application also provides a data processing device based on a containerized application cluster management platform, comprising:

[0059] An acquisition unit, used to acquire data set information created based on a containerized application cluster management platform;

[0060] An orchestration unit, configured to perform storage orchestration on the data set according to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, and obtain a set of schedulable computing nodes capable of storing the data set;

[0061] A preloading unit is used to preload the data in the data set into the schedulable computing node, on a target computing node of the storage space that matches the occupied space of the data.

[0062] The present application also provides an application scheduling method based on a containerized application cluster management platform, including:

[0063] Based on the containerized application cluster management platform, determine the data set information that the application service needs to use;

[0064] Based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and orchestrated to obtain a set of schedulable computing nodes capable of storing the data set, and the scheduling information of the application service is determined;

[0065] According to the scheduling information, the application service is scheduled to a matching target computing node in the set of schedulable computing nodes.

[0066] In some embodiments, the determining of the data set information required for the application service based on the containerized application cluster management platform includes:

[0067] Based on the containerized application cluster management platform, the data set information to be used is specified in a configuration file for creating the application service.

[0068] In some embodiments, the determining the scheduling information of the application service based on the storage arrangement of the data set according to the occupied space of the data set in the data set information and the storage space of the computing node in the containerized application cluster management platform, and obtaining a set of schedulable computing nodes capable of storing the data set, includes:

[0069] Based on the set of schedulable computing nodes, configure selection configuration information that meets the node selection condition and path configuration information for accessing the computing nodes that meet the node selection condition;

[0070] The configuration file including the selection configuration information and the path configuration information is determined as the scheduling information of the application service.

[0071] In some embodiments, configuring selection configuration information satisfying a node selection condition and path configuration information for accessing the computing nodes satisfying the node selection condition based on the set of schedulable computing nodes includes:

[0072] Based on the created configuration file of the application service, the selection configuration information of the configuration satisfying the node selection condition is added, and the path configuration information of the access to the computing node satisfying the node selection condition is added.

[0073] In some embodiments, scheduling the application service to a matching target computing node in the set of schedulable computing nodes according to the scheduling information includes:

[0074] Determine the computing node that meets the node selection condition in the scheduling information as the target computing node;

[0075] The application service is scheduled according to the access path of the target computing node in the scheduling information.

[0076] In some embodiments, it also includes:

[0077] Sorting the determined target computing nodes according to the storage capacity of storing the data set;

[0078] The step of scheduling the application service according to the access path of the target computing node in the scheduling information includes:

[0079] The application service is scheduled according to the access path of the target computing node with the largest storage capacity within the sorted target computing node range.

[0080] The present application also provides an application scheduling device based on a containerized application cluster management platform, comprising:

[0081] A first determination unit is used to determine data set information that the application service needs to use based on the containerized application cluster management platform;

[0082] A second determining unit is used to determine the scheduling information of the application service by performing storage orchestration on the data set based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, and obtaining a set of schedulable computing nodes capable of storing the data set;

[0083] The scheduling unit is used to schedule the application service to a matching target computing node in the set of schedulable computing nodes according to the scheduling information.

[0084] The present application also provides a computer storage medium for storing data generated by a network platform, and a program for processing the data generated by the network platform;

[0085] When the program is read and executed, the following operations are performed:

[0086] Obtain the dataset information created based on the containerized application cluster management platform;

[0087] According to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set;

[0088] Preloading the data in the data set into the set of schedulable computing nodes, onto a target computing node of the storage space that matches the occupied space of the data;

[0089] or,

[0090] Based on the containerized application cluster management platform, determine the data set information that the application service needs to use;

[0091] Based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and orchestrated to obtain a set of schedulable computing nodes capable of storing the data set, and the scheduling information of the application service is determined;

[0092] According to the scheduling information, the application service is scheduled to a matching target computing node in the set of schedulable computing nodes.

[0093] The present application also provides an electronic device, comprising:

[0094] processor;

[0095] The memory is used to store a program for processing data generated by the network platform, and when the program is read and executed by the processor, the program performs the following operations:

[0096] Obtain the dataset information created based on the containerized application cluster management platform;

[0097] According to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set;

[0098] Preloading the data in the data set into the set of schedulable computing nodes, onto a target computing node of the storage space that matches the occupied space of the data;

[0099] or,

[0100] Based on the containerized application cluster management platform, determine the data set information that the application service needs to use;

[0101] Based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and orchestrated to obtain a set of schedulable computing nodes capable of storing the data set, and the scheduling information of the application service is determined;

[0102] According to the scheduling information, the application service is scheduled to a matching target computing node in the set of schedulable computing nodes.

[0103] Compared with the prior art, this application has the following advantages:

[0104] The present application provides an embodiment of a data processing method based on a containerized application cluster management platform, by obtaining data set information created based on a containerized application cluster management platform; according to the occupied space of the data set in the data set information and the storage space of the computing node in the containerized application cluster management platform, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set; the data in the data set is preloaded into the set of schedulable computing nodes, on the target computing node of the storage space that matches the occupied space of the data. In this way, the data required by the application service can be extracted from the storage node in the form of a data set and stored on the computing node of the specified availability zone, so that the application service can directly access the data stored on the computing node when running, without requesting to access the data from the storage node, avoiding the delay problem caused by the application service needing to request to access the data from the storage node when executing, and improving the efficiency of subsequent data analysis and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1 It is a flow chart of an embodiment of a data processing method based on a containerized application cluster management platform provided by the present application;

[0106] Figure 2 It is a structural schematic diagram of an embodiment of a data processing device based on a containerized application cluster management platform provided by the present application;

[0107] Figure 3 It is a flow chart of an embodiment of an application scheduling method based on a containerized application cluster management platform provided by the present application;

[0108] Figure 4 It is a structural schematic diagram of an embodiment of an application scheduling device based on a containerized application cluster management platform provided by the present application;

[0109] Figure 5 It is a structural schematic diagram of an electronic device embodiment provided by the present application. DETAILED DESCRIPTION

[0110] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0111] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The descriptions used in this application and the appended claims, such as "a", "first", and "second", are not limitations on quantity or sequence, but are used to distinguish the same type of information from each other.

[0112] Based on the description of the above background technology, it can be seen that when managing containerized applications, the efficiency of data processing and analysis will be affected due to the problem of data access delay. To address this problem, the prior art provides two solutions, as follows:

[0113] 1. Use high iops (Input / Output Operations Per Second: an indicator for measuring the performance of storage devices or storage networks) distributed data storage methods such as CPFS (Cloud Paralleled File System). CPFS data is stored in multiple data nodes in the cluster and can be accessed by multiple clients simultaneously, thereby providing high IOPS, high throughput, and low latency data storage services for large high-performance computer clusters. However, the shortcomings of this solution are: high economic cost pressure and complex data synchronization operations due to the large data scale.

[0114] 2. Using Alluxio (memory high-speed virtual distributed storage system) and other caching technologies that are currently popular in the industry, they essentially use themselves as agents for application data access, caching remote data locally while acquiring data, and when local storage space is insufficient, cached data needs to be evicted. This data caching strategy lacks intelligence combined with applications. For example, when an application runs on a node for the first time, it has accumulated a certain amount of cached data; and there is no mechanism to ensure that the application can be scheduled to the node again at the next scheduling time.

[0115] It can be seen that the above two solutions cannot fundamentally improve the efficiency of data processing and analysis. Especially in the face of a large number of AI customer model training and high-performance computing scenarios, the data access efficiency determines the throughput of model training or data processing performance.

[0116] In view of the above-mentioned defects of the prior art, the present application provides a data processing method embodiment based on a containerized application cluster management platform, which can provide an effective solution to the data access delay problem in the prior art.

[0117] Please refer to Figure 1 As shown, Figure 1 This is a flow chart of an embodiment of a data processing method based on a containerized application cluster management platform provided by the present application. The embodiment of the method includes:

[0118] Step S101: Acquire data set information created based on the containerized application cluster management platform;

[0119] The purpose of step S101 is to obtain data set information. In this embodiment, the data set information can be created by a user. Of course, other creation methods other than the user are not excluded, such as automatic creation of data set information. The automatic creation of data set information can be based on a set creation trigger condition to achieve automatic creation of the data set information. The creation trigger condition can be based on at least one of the use frequency of application data, the use amount of application data, and the use time of application data.

[0120] The data set information may include a data set required by the application service, and other information used to describe the relevant content of the data set. In this embodiment, the containerized application cluster management platform may be Kubernetes, referred to as K8s, that is, managing containerized applications on multiple hosts in the cloud platform. The goal of Kubernetes is to make the deployment of containerized applications simple and efficient (powerful). Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance. In this embodiment, K8s is used as an example. In fact, the containerized application cluster management platform can also be used in other ways.

[0121] In this embodiment, the user creates data set information as an example for description. Therefore, when obtaining the data set information, it is necessary to create the data set information in advance. Specifically, the data set information can be created by the user through a configuration file based on the containerized application cluster management platform. In this embodiment, the yaml configuration file writing language can be used to define the data set information in the configuration file, that is, configure or define the following information in the configuration file:

[0122] apiVersion (specify api version): "data-orchestration.org / v1aphla1";

[0123] kind (specify the role / type of the resource to be created): "Dataset" (dataset);

[0124] metadata (metadata / attributes of the resource):

[0125] name (the name of the resource): "imagenet";

[0126] spec (the content of the resource):

[0127] pv (Persitent Volume: specified storage): imagenet-oss;

[0128] path (the resource storage path or the dataset storage path): / imagenet;

[0129] selector (node ​​selection):

[0130] Az (Availabe Zone): hangzhou-b (Hangzhou-b);

[0131] GPU(specified model):true.

[0132] The above content describes how to configure the dataset and related information of the dataset in the configuration file. In the configuration file described above, the user creates the dataset information including: defining the dataset, defining the name of the dataset, defining the storage address of the dataset, defining the available area of ​​the dataset, and defining the specified model of the dataset. Of course, the related information of the dataset can be part of the above listed information or all of it, or other required information beyond the listed information.

[0133] From the configuration of the above configuration file, we know that users need to define PersitentVolume in the configuration file to specify the storage method of the data set. For example, you can specify that the data set is stored in at least one of NAS, OSS, and S3. In the above configuration file example, the storage node specified by the data set Dataset is OSS, the specified computer model is GPU, and the selected computing node availability zone is hangzhou-b.

[0134] In this embodiment, the purpose of creating data set information is to extract the data set required by the application service from the storage node. The data set can be placed on the computing node in the available zone, thereby speeding up data access and avoiding data access delays.

[0135] Therefore, based on the above step S101, it is necessary to arrange how the data set in the data set information is stored on the computing nodes in the available zone, that is, to execute the above step S102.

[0136] Step S102: According to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and orchestrated to obtain a set of schedulable computing nodes capable of storing the data set.

[0137] The purpose of step S102 is to determine how to store the data set on the computing nodes in the available zone. It should be noted that the available zone here can be a computing node cluster in a specified area that can store data in the data set.

[0138] For the computing node available zone selected in the data set information created in step S101, the data in the data set is stored and arranged in the available zone.

[0139] It is understandable that in a computing node cluster in an availability zone, not all computing nodes in the cluster can store the data in the dataset, because the storage space size of the computing node is not fixed. Therefore, the storage of the dataset needs to be arranged according to the space occupied by the dataset and the storage space of the computing node.

[0140] In this embodiment, the storage space of the computing node may be a cache space of the computing node.

[0141] The specific implementation process of step S102 may include:

[0142] Step S102-1-11: querying the occupied space of the data set according to the storage node specified in the data set information;

[0143] In this embodiment, in step S102 - 1 , the occupied space of the dataset (Dataset) can be queried from the designated storage node OSS through the dataset scheduler.

[0144] Step S102-1-12: Selecting candidate computing nodes that meet the screening requirements in the containerized application cluster management platform;

[0145] In this embodiment, the specific implementation process of step S102-2 may include selecting a candidate computing node in the following manner:

[0146] In the containerized application cluster management platform, selecting the candidate computing node that matches the label information defined in the data set information according to the label information of the computing node (i.e., node label);

[0147] or,

[0148] Selecting the candidate computing node that meets the availability zone range requirement defined in the data set information in the containerized application cluster management platform;

[0149] or,

[0150] The candidate computing node that meets the specified device type defined in the data set information is selected in the containerized application cluster management platform. For example, it can be the processor type of the specified computing node, such as GPU or CPU, and of course it can also be other types of information that can characterize the computing node type.

[0151] Step S102-1-13: According to the occupied space information of the data set and the storage space of the candidate computing nodes, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set.

[0152] Considering that there is no computing node in the containerized application cluster that can store all of the data sets, in other words, when the storage space of the computing node is insufficient to store the data sets, the present embodiment may further include:

[0153] According to the space occupied by the data set or the usage requirement of the data set, the data set is segmented to obtain at least two data set blocks;

[0154] Recording the name of the data set block and the amount of the data set block corresponding to the name that needs to be allocated to the containerized application cluster into the created data set record object;

[0155] In this embodiment, the segmentation of the data set is mainly explained by taking the occupied space of the data set as an example, and specifically, the data set may be segmented according to the occupied space of files included in the data set.

[0156] The specific implementation process of step S102-1-13 may be: according to the occupied space of the data set blocks in the data set record object and the storage space of the candidate computing nodes, the data set is stored and arranged to obtain a set of schedulable computing nodes that can store the data set.

[0157] To facilitate storage arrangement of the data set, this embodiment may further include:

[0158] A computing node list for recording candidate computing node information is created according to the selected candidate computing nodes; the candidate computing node information includes: identification information of the candidate computing nodes and storage space information of the candidate computing nodes.

[0159] The specific implementation process of step S102-1-13 may be: according to the space occupied by the data set in the data set record object and the storage space of the candidate computing nodes in the computing node list, the data set is stored and arranged to obtain a set of schedulable computing nodes that can store the data set.

[0160] It can be learned from the above step S101 that when creating a data set, the available area of ​​the computing node where the data set is stored can be specified. For example, if the available area specified in the above is Hangzhou-b, then the computing node cluster is all computing nodes in the Hangzhou-b area. In this embodiment, the computing node list can be created in two ways:

[0161] Method 1 is:

[0162] Sorting the candidate computing nodes in ascending or descending order according to the storage capacity of the available storage space;

[0163] The computing node list is created according to the sorted candidate computing nodes.

[0164] Method 2 is:

[0165] Sorting the candidate computing nodes in ascending or descending order according to the storage capacity of the available storage space;

[0166] The sorted information of the candidate computing nodes is recorded in a pre-created computing node list.

[0167] In this embodiment, for the convenience of subsequently adjusting the application service to the computing node, the computing node list can be stored in descending order, that is, the candidate computing node list in the available area cluster can be arranged from large to small according to the available storage space.

[0168] The specific storage arrangement process in step S102 may include:

[0169] Step S102-2-11: Determine whether the computing node list has a set of available candidate computing nodes that meet the set available threshold requirement; wherein the available threshold requirement may be that the storage space of the candidate computing node is greater than zero, that is, it is capable of storing data. When the available storage space of the candidate computing node in the computing node list is greater than zero, the candidate computing node is an available candidate computing node. The available candidate computing node set includes: at least one available candidate computing node.

[0170] Step S102-2-12: If yes, determine whether the space occupied by the data set is less than the storage capacity of the storage space of the available candidate computing nodes in the available candidate computing node set;

[0171] It can be understood that if the determination result of step S102-2-11 is no, it means that there are no candidate computing nodes in the computing node list that can store the data set. At this time, you can exit the storage arrangement of the data set, that is, exit the step of storing and arranging the data set according to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform to obtain a set of schedulable computing nodes that can store the data set; re-execute the step of selecting candidate computing nodes that meet the screening requirements in the containerized application cluster management platform; or wait according to the set waiting time. When the upper limit of the waiting time is reached, it can be re-determined whether there is a set of available candidate computing nodes in the computing node list that meets the set available threshold requirements. In other words, there can be multiple execution methods after exiting, and they are not limited to the above content.

[0172] It should be noted that the dataset can determine whether there is an available candidate computing node set in the computing node list based on the occupied space in the form of a complete dataset, or it can determine whether there is an available candidate computing node set in the computing node list in the form of dataset blocks. In the process of determining in the form of dataset blocks, it is necessary to determine in sequence whether there are available candidate computing nodes in the computing node list with a storage capacity greater than zero according to the dataset blocks. Regardless of the form, if there are no available candidate computing nodes with a storage capacity greater than zero, it means that there are no candidate computing nodes in the computing node list that can store the dataset.

[0173] Step S102-2-13: If the determination result of step S102-2-12 is yes, selecting an available candidate computing node that matches the occupancy in the available candidate computing node set;

[0174] When the determining whether the occupied space of the data set is less than the storage space of the available candidate computing node storage space in the available candidate computing node set is no, the method may include:

[0175] Determine whether the space occupied by the data set is greater than the storage capacity of the storage space of the available candidate computing nodes in the set of available candidate computing nodes;

[0176] If yes, selecting the available candidate computing node with the largest storage capacity from the range of available candidate computing nodes;

[0177] The amount of space occupied by the data set is updated, and the process returns to the step of determining whether the computing node list has a set of available candidate computing nodes that meets the set availability threshold requirement;

[0178] If the answer is no when determining whether the space occupied by the data set is greater than the storage capacity of the storage space of the available candidate computing nodes in the set of available candidate computing nodes, then an available candidate computing node with the same storage capacity as the occupied space is selected from the range of available candidate computing nodes.

[0179] Step S102-2-14: Determine the available candidate computing nodes as the set of schedulable computing nodes that can store the data set.

[0180] The specific implementation process of step S102-2-14 can be determined according to different determination results in step S102-2-13, as follows:

[0181] Based on step S102-2-13, the available candidate computing nodes are determined as the set of schedulable computing nodes that can store the data set.

[0182] Based on selecting the available candidate computing node with the largest storage capacity from the range of the available candidate computing nodes, the step S102-2-14 may be to determine the selected available candidate computing node with the largest storage capacity as the set of schedulable computing nodes that can store the data set.

[0183] Based on selecting the available candidate computing nodes whose storage capacity is equal to the occupancy capacity from the range of the available candidate computing nodes, the step S102-2-14 may be to determine the selected available candidate computing nodes whose storage capacity is equal to the occupancy capacity as the set of schedulable computing nodes.

[0184] In this embodiment, the step S102-2-14 may record the available candidate computing nodes into a scheduling list, where the scheduling list is used to record information about the set of schedulable computing nodes that can store the data set.

[0185] Step S103: preloading the data in the data set into the set of schedulable computing nodes, onto a target computing node of the storage space that matches the occupied space of the data.

[0186] It is understandable that in application scenarios such as artificial intelligence model training or machine learning, the data stored in the storage node will be updated in real time. In order to ensure the synchronization of the created data set in the computing node cluster in the specified availability zone, the changes in the storage node data can be monitored. Once the storage node data changes, the step S102 will be re-executed. The data change here can be a change in data growth.

[0187] The changed data set can also be re-segmented, and the names of the re-segmented data set blocks and the occupancy of the data set blocks corresponding to the names that need to be allocated to the containerized application cluster can be recorded in the data set record object, overwriting the previous record content. Of course, it can also be recorded in the re-created data set record object.

[0188] For the subsequent application scheduling to the computing node cluster as a reference, in this embodiment, the following may also be included:

[0189] Acquire at least one of storage capacity information of the data set in the set of schedulable computing nodes and access capacity information of data in the data set stored in the set of schedulable computing nodes;

[0190] The information is used as reference information for application scheduling in the containerized application cluster management platform.

[0191] The specific methods are as follows:

[0192] apiVersion:"data-orchestration.org / v1aphla1"

[0193] kind:"Dataset"

[0194] metadata:

[0195] name:"imagenet"

[0196] spec:

[0197] pv:imagenet-oss

[0198] path: / imagenet

[0199] status (storage status):

[0200] total (total data usage): 50GiB

[0201] cached (stored storage): 15GiB

[0202] Cacheable (storage capacity supported): 40GiB

[0203] needMoreForCache (the amount of storage needed to be expanded): 10GiB.

[0204] The above storage capacity is used as an example for explanation. According to the storage capacity, it can be known whether it is necessary to expand the capacity of the schedulable computing node set to ensure that all data in the data set can be stored on the matching target node.

[0205] The above is a description of an embodiment of a data processing method based on a containerized application cluster management platform provided by the present application. This embodiment can extract the data required by the application service from the storage node in the form of a data set and store it on the computing node of the specified availability zone, thereby ensuring that the application service can directly access the data stored on the computing node during runtime without requesting access to the data from the storage node, avoiding the delay problem of the computing node requesting access to the data from the storage node, and improving the efficiency of data analysis and processing.

[0206] The above is a specific description of an embodiment of a data processing method based on a containerized application cluster management platform provided by the present application. Corresponding to the aforementioned embodiment of a data processing method based on a containerized application cluster management platform, the present application also discloses an embodiment of a data processing device based on a containerized application cluster management platform. Please refer to Figure 2 Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0207] like Figure 2 As shown, Figure 2 : is a structural schematic diagram of an embodiment of a data processing device based on a containerized application cluster management platform provided by the present application, and the embodiment of the device includes:

[0208] An acquisition unit 201 is used to acquire data set information created based on a containerized application cluster management platform;

[0209] The acquisition unit 201 is specifically configured to acquire the created data set information according to an application orchestration configuration file defined based on a containerized application cluster management platform.

[0210] In this embodiment, it is also necessary to create the data set information, so it may also include:

[0211] The creation unit is used to define the data set information based on the configuration file of the containerized application cluster management platform. In this embodiment, the yaml configuration file writing language can be used to define the data set information in the configuration file. Please refer to the description of step S101 above for details, which will not be repeated here.

[0212] The orchestration unit 202 is used to perform storage orchestration on the data set according to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, and obtain a set of schedulable computing nodes capable of storing the data set;

[0213] The arrangement unit 202 includes: a query subunit, a selection subunit and an arrangement subunit;

[0214] The query subunit is specifically used to query the occupied space of the data set according to the storage end specified in the data set information;

[0215] The selection subunit is specifically used to select candidate computing nodes that meet the screening requirements in the containerized application cluster management platform;

[0216] The arrangement subunit is specifically configured to perform storage arrangement on the data set according to the occupied space information of the data set and the storage space of the candidate computing nodes, so as to obtain a set of schedulable computing nodes capable of storing the data set.

[0217] The selection subunit may adopt at least one of the following selection methods when selecting the candidate computing node:

[0218] In the containerized application cluster management platform, selecting the candidate computing node that matches the label information defined in the data set information according to the label information of the computing node;

[0219] Selecting the candidate computing node that meets the availability zone range requirement defined in the data set information in the containerized application cluster management platform;

[0220] The candidate computing node that meets the specified device type defined in the data set information is selected in the containerized application cluster management platform.

[0221] In this embodiment, the arrangement unit may further include: a segmentation subunit and an object creation subunit.

[0222] The segmentation subunit is used to segment the data set according to the occupied space of the data set to obtain at least two data set blocks;

[0223] The object creation subunit is used to record the name of the data set block and the occupancy of the data set block corresponding to the name and required to be allocated to the containerized application cluster into the created data set record object;

[0224] The arrangement subunit is specifically used to perform storage arrangement on the data set according to the occupied space of the data set blocks in the data set record object and the storage space of the candidate computing nodes, so as to obtain a set of schedulable computing nodes capable of storing the data set.

[0225] In this embodiment, the method may further include: a node list creation subunit, which is used to create a computing node list for recording candidate computing node information according to the selected candidate computing nodes.

[0226] The arrangement subunit is specifically used to perform storage arrangement on the data set according to the space occupied by the data set in the data set record object and the storage space of the candidate computing node in the computing node list, so as to obtain a set of schedulable computing nodes capable of storing the data set.

[0227] The node list creation subunit includes: a sorting subunit, which is used to sort the candidate computing nodes in ascending order or descending order according to the storage capacity information of the available storage space.

[0228] The node list creation subunit is specifically used to create the computing node list according to the candidate computing nodes sorted in the sorting subunit.

[0229] The arrangement subunit includes: a first determination subunit, a second determination subunit, a selection subunit and a third determination subunit.

[0230] The first determining subunit is used to determine whether the computing node list has a set of available candidate computing nodes that meets a set availability threshold requirement;

[0231] The second determining subunit is used to determine whether the occupied space of the data set is less than the storage capacity of the storage space of the available candidate computing nodes in the available candidate computing node set when the determination result of the first determining subunit is yes;

[0232] The selecting subunit is configured to select, when the determination result in the second determining subunit is yes, an available candidate computing node matching the occupancy in the available candidate computing node set;

[0233] The third determining subunit is used to determine the available candidate computing nodes selected in the selecting subunit as the set of schedulable computing nodes that can store the data set.

[0234] When the determination result of the first determination subunit is no, it includes an exit subunit for exiting the orchestration subunit; or a return subunit for returning to the selection subunit to re-execute the selection of candidate computing nodes that meet the screening requirements in the containerized application cluster management platform; or a waiting subunit for waiting according to a set waiting time, because the candidate computing node may release storage space according to the current operating conditions, and therefore, it is possible to wait according to the set waiting time.

[0235] When the determination result of the second determination subunit is no, it also includes: a fourth determination subunit, used to determine whether the amount of space occupied by the data set is greater than the storage capacity of the storage space of the available candidate computing nodes in the set of available candidate computing nodes; the selection subunit, based on the determination result of the fourth determination subunit is yes, is used to select the available candidate computing node with the largest storage capacity from the range of available candidate computing nodes; an update subunit, used to update the amount of space occupied by the data set, and return to the first determination subunit to continue execution until all candidate computing nodes in the computing node list are determined; the third determination subunit is specifically used to determine the selected available candidate computing node with the largest storage capacity as the set of schedulable computing nodes that can store the data set.

[0236] When the determination result of the fourth determination subunit is no, it means that the amount of space occupied by the data set is equal to the storage amount of the storage space of the available candidate computing nodes. At this time, the selection subunit is specifically used to select the available candidate computing nodes whose storage amount is equal to the occupied amount from the range of the available candidate computing nodes; the third determination subunit is specifically used to determine the selected available candidate computing nodes whose storage amount is equal to the occupied amount as the set of schedulable computing nodes.

[0237] The third determining subunit includes: a recording subunit, configured to record the available candidate computing nodes into a scheduling list, wherein the scheduling list is configured to record information of the set of schedulable computing nodes capable of storing the data set.

[0238] The preloading unit 203 is used to preload the data in the data set into the schedulable computing node, on the target computing node of the storage space that matches the occupied space of the data.

[0239] In this embodiment, it can also include: a data change determination unit, which is used to determine whether the space occupied by the data set on the storage terminal has changed during the storage arrangement of the data set based on the data set occupied space information in the data set record object and the storage space of the candidate computing node; the creation subunit can specifically recreate the data record object for recording the space occupied by the data set according to the data set after the storage amount changes when the determination result of the data change determination subunit is yes.

[0240] In this embodiment, it may also include: an information acquisition unit and a reference information determination unit, the information acquisition unit is used to obtain at least one of the storage capacity information of the data set in the set of schedulable computing nodes and the access capacity information of the data stored in the data set in the set of schedulable computing nodes; the reference information determination unit is used to use the information as reference information for application scheduling in the containerized application cluster management platform.

[0241] The above is a brief description of an embodiment of a data processing device based on a containerized application cluster management platform provided by the present application. For the content of the device embodiment, reference can be made to the content of the above method embodiment, and no detailed repetition will be given here.

[0242] Based on the above content, this application also provides an application scheduling method based on a containerized application cluster management platform. This method mainly provides a method of scheduling applications to matching computing nodes based on the data processing method of the above containerized application cluster management platform, so that the data required by the application service at runtime can be directly provided by the computing node. For the specific implementation process, please refer to Figure 3 , Figure 3 : is a flowchart of an embodiment of an application scheduling method based on a containerized application cluster management platform provided by the present application, and the scheduling method embodiment includes:

[0243] Step S301: Determine the data set information required by the application service based on the containerized application cluster management platform;

[0244] The specific implementation process of step S301 may include:

[0245] Based on the containerized application cluster management platform, the data set information to be used is specified in the configuration file for creating the application service. The data information may include information such as the specified data set name, data set content, data set address, and data set storage location. The specific configuration file content may be as follows:

[0246]

[0247]

[0248] Based on the above content, it can be understood that the data set information required by the application service can be specified by annotations in the configuration file of the containerized application cluster management platform when creating the application. Of course, the configuration file can also be modified during the operation of the application service to perform corresponding operations.

[0249] Step S302: Based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and orchestrated, a set of schedulable computing nodes capable of storing the data set is obtained, and scheduling information of the application service is determined;

[0250] The specific implementation process of step S302 may include:

[0251] Step S302-1: Based on the set of schedulable computing nodes, configure selection configuration information that meets the node selection condition and path configuration information for accessing the computing nodes that meet the node selection condition;

[0252] Step S302-2: Determine the configuration file including the selection configuration information and the path configuration information as the scheduling information of the application service.

[0253] The specific implementation process of step S302 may be: based on the created configuration file of the application service, adding the selection configuration information that satisfies the node selection condition, and adding the path configuration information for accessing the computing node that satisfies the node selection condition.

[0254] Based on the example of the above configuration file, the additions are as follows:

[0255]

[0256]

[0257] Step S303: according to the scheduling information, the application service is scheduled to a matching target computing node in the set of schedulable computing nodes.

[0258] The specific implementation process of step S303 may include:

[0259] The computing node that meets the node selection condition in the scheduling information is determined as the target computing node; in this embodiment, the node selection condition can be to select the computing node that is closest to the application service as the target computing node, or to select the computing node with the largest data access volume as the target computing node, or to select the computing node with the largest amount of data stored as the target computing node.

[0260] The application service is scheduled according to the access path of the target computing node in the scheduling information.

[0261] In this embodiment, the scheduler plugin in K8s can be used to sort the target computing nodes according to the storage capacity of the storage data sets when scheduling application services to the target computing nodes. The higher the storage capacity, the higher the possibility that the application service will be scheduled to the target computing node with high storage capacity.

[0262] Therefore, this embodiment may further include:

[0263] Sorting the determined target computing nodes according to the storage capacity of storing the data set;

[0264] The step of scheduling the application service according to the access path of the target computing node in the scheduling information includes:

[0265] The application service is scheduled according to the access path of the target computing node with the largest storage capacity within the sorted target computing node range.

[0266] The above is a description of an embodiment of an application scheduling method based on a containerized application cluster management platform provided by the present application. Corresponding to the embodiment of an application scheduling method based on a containerized application cluster management platform provided above, the present application also discloses an embodiment of an application scheduling device based on a containerized application cluster management platform, please refer to Figure 4 Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0267] The application scheduling device embodiment includes:

[0268] The first determination unit 401 is used to determine the data set information required by the application service based on the containerized application cluster management platform; the specific implementation process of the first determination unit 401 can refer to the description of the above step S301.

[0269] The second determination unit 402 is used to perform storage orchestration on the data set based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, to obtain a set of schedulable computing nodes capable of storing the data set, and to determine the scheduling information of the application service; the specific implementation process of the second determination unit 402 can refer to the description of the above step S302.

[0270] The scheduling unit 403 is used to schedule the application service to a matching target computing node in the set of schedulable computing nodes according to the scheduling information. The specific implementation process of the scheduling unit 403 can refer to the description of the above step S303.

[0271] Based on the above content, the present application also provides a computer storage medium for storing data generated by a network platform, and a program for processing the data generated by the network platform;

[0272] When the program is read and executed, the following operations are performed:

[0273] Obtain the dataset information created based on the containerized application cluster management platform;

[0274] According to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set;

[0275] Preloading the data in the data set into the set of schedulable computing nodes, onto a target computing node of the storage space that matches the occupied space of the data;

[0276] The step of obtaining the data set information created based on the containerized application cluster management platform includes:

[0277] According to the application orchestration configuration file defined based on the containerized application cluster management platform, the created data set information is obtained.

[0278] The step of performing storage arrangement on the data set according to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform to obtain a set of schedulable computing nodes capable of storing the data set includes:

[0279] According to the storage end specified in the data set information, query the occupied space of the data set;

[0280] Selecting candidate computing nodes that meet the screening requirements in the containerized application cluster management platform;

[0281] According to the occupied space information of the data set and the storage space of the candidate computing node, the data set is stored and arranged to obtain a set of schedulable computing nodes that can store the data set.

[0282] The step of selecting a candidate computing node that meets the screening requirements in the containerized application cluster management platform includes at least one of the following selection methods:

[0283] In the containerized application cluster management platform, selecting the candidate computing node that matches the label information defined in the data set information according to the label information of the computing node;

[0284] Selecting the candidate computing node that meets the availability zone range requirement defined in the data set information in the containerized application cluster management platform;

[0285] The candidate computing node that meets the specified device type defined in the data set information is selected in the containerized application cluster management platform.

[0286] Among them, it also includes:

[0287] According to the space occupied by the data set or the usage requirement of the data set, the data set is segmented to obtain at least two data set blocks;

[0288] Recording the name of the data set block and the amount of the data set block corresponding to the name that needs to be allocated to the containerized application cluster into the created data set record object;

[0289] The step of performing storage arrangement on the data set according to the occupied space information of the data set and the storage space of the candidate computing node to obtain a set of schedulable computing nodes capable of storing the data set includes:

[0290] According to the occupied space of the data set blocks in the data set record object and the storage space of the candidate computing nodes, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set.

[0291] Among them, it also includes:

[0292] Creating a computing node list for recording candidate computing node information according to the selected candidate computing nodes;

[0293] The step of performing storage arrangement on the data set according to the space occupied by the data set in the data set record object and the storage space of the candidate computing node to obtain a set of schedulable computing nodes capable of storing the data set includes:

[0294] According to the space occupied by the data set in the data set record object and the storage space of the candidate computing node in the computing node list, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set.

[0295] The step of creating a computing node list for recording information of the candidate computing nodes according to the selected candidate computing nodes includes:

[0296] Sorting the candidate computing nodes in ascending or descending order according to the storage capacity information of the available storage space;

[0297] The computing node list is created according to the sorted candidate computing nodes.

[0298] The step of performing storage arrangement on the data set according to the space occupied by the data set in the data set record object and the storage space of the candidate computing node to obtain a set of schedulable computing nodes capable of storing the data set includes:

[0299] Determine whether the computing node list has a set of available candidate computing nodes that meets a set availability threshold requirement;

[0300] If so, determining whether the space occupied by the data set is less than the storage capacity of the storage space of the available candidate computing nodes in the set of available candidate computing nodes;

[0301] If so, selecting an available candidate computing node that matches the occupancy from the set of available candidate computing nodes;

[0302] The available candidate computing nodes are determined as the set of schedulable computing nodes that can store the data set.

[0303] Among them, it also includes:

[0304] When the determination of whether the occupied amount of the space occupied by the data set is less than the storage amount of the storage space of the available candidate computing nodes in the set of available candidate computing nodes is no, then determining whether the occupied amount of the space occupied by the data set is greater than the storage amount of the storage space of the available candidate computing nodes in the set of available candidate computing nodes;

[0305] If yes, selecting the available candidate computing node with the largest storage capacity from the range of available candidate computing nodes;

[0306] The amount of space occupied by the data set is updated, and the process returns to the step of determining whether the computing node list has a set of available candidate computing nodes that meets the set availability threshold requirement;

[0307] The step of determining the available candidate computing nodes as the set of schedulable computing nodes capable of storing the data set comprises:

[0308] The selected available candidate computing node with the largest storage capacity is determined as the set of schedulable computing nodes that can store the data set.

[0309] Among them, it also includes:

[0310] When determining whether the occupied space of the data set is greater than the storage capacity of the storage space of the available candidate computing nodes in the set of available candidate computing nodes, if the result is no, selecting an available candidate computing node with the storage capacity equal to the occupied space from the range of the available candidate computing nodes;

[0311] The step of determining the available candidate computing nodes as the set of schedulable computing nodes capable of storing the data set comprises:

[0312] The selected available candidate computing nodes whose storage capacity is equal to the occupied capacity are determined as the schedulable computing node set.

[0313] The step of determining the available candidate computing nodes as the set of schedulable computing nodes capable of storing the data set includes:

[0314] The available candidate computing nodes are recorded in a scheduling list, where the scheduling list is used to record information about the set of schedulable computing nodes that can store the data set.

[0315] Among them, it also includes:

[0316] When the determination of whether there is a set of available candidate computing nodes that meet the set availability threshold requirements in the computing node list is no, exit the step of performing storage arrangement for the data set according to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform to obtain a set of schedulable computing nodes that can store the data set, or re-execute the step of selecting candidate computing nodes that meet the screening requirements in the containerized application cluster management platform, or wait according to the set waiting time.

[0317] Among them, it also includes:

[0318] When performing storage arrangement for the data set according to the space occupied by the data set in the data set record object and the storage space of the candidate computing node, determining whether the space occupied by the data set on the storage terminal changes;

[0319] If so, the data record object for recording the space occupied by the data set is recreated according to the data set after the storage amount is changed.

[0320] Among them, it also includes:

[0321] Acquire at least one of storage capacity information of the data set in the set of schedulable computing nodes and access capacity information of data in the data set stored in the set of schedulable computing nodes;

[0322] The information is used as reference information for application scheduling in the containerized application cluster management platform.

[0323] Alternatively, when the program is read and executed, the following operations are performed:

[0324] Based on the containerized application cluster management platform, determine the data set information that the application service needs to use;

[0325] Based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and orchestrated to obtain a set of schedulable computing nodes capable of storing the data set, and the scheduling information of the application service is determined;

[0326] According to the scheduling information, the application service is scheduled to a matching target computing node in the set of schedulable computing nodes.

[0327] The method of determining the data set information required for the application service based on the containerized application cluster management platform includes:

[0328] Based on the containerized application cluster management platform, the data set information to be used is specified in a configuration file for creating the application service.

[0329] Wherein, based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and arranged, and a set of schedulable computing nodes capable of storing the data set is obtained, and the scheduling information of the application service is determined, including:

[0330] Based on the set of schedulable computing nodes, configure selection configuration information that meets the node selection condition and path configuration information for accessing the computing nodes that meet the node selection condition;

[0331] The configuration file including the selection configuration information and the path configuration information is determined as the scheduling information of the application service.

[0332] The configuring, based on the set of schedulable computing nodes, the selection configuration information satisfying the node selection condition and the path configuration information for accessing the computing nodes satisfying the node selection condition includes:

[0333] Based on the created configuration file of the application service, the selection configuration information of the configuration satisfying the node selection condition is added, and the path configuration information of the access to the computing node satisfying the node selection condition is added.

[0334] The step of scheduling the application service to a matching target computing node in the set of schedulable computing nodes according to the scheduling information includes:

[0335] Determine the computing node that meets the node selection condition in the scheduling information as the target computing node;

[0336] The application service is scheduled according to the access path of the target computing node in the scheduling information.

[0337] Among them, it also includes:

[0338] Sorting the determined target computing nodes according to the storage capacity of storing the data set;

[0339] The step of scheduling the application service according to the access path of the target computing node in the scheduling information includes:

[0340] The application service is scheduled according to the access path of the target computing node with the largest storage capacity within the sorted target computing node range.

[0341] Based on the above content, the present application also provides an electronic device, such as Figure 5 As shown, the electronic device embodiment includes:

[0342] Processor 501;

[0343] The memory 502 is used to store a program for processing data generated by the network platform. When the program is read and executed by the processor, the following operations are performed:

[0344] Obtain the dataset information created based on the containerized application cluster management platform;

[0345] According to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set;

[0346] Preloading the data in the data set into the set of schedulable computing nodes, onto a target computing node of the storage space that matches the occupied space of the data;

[0347] The step of obtaining the data set information created based on the containerized application cluster management platform includes:

[0348] According to the application orchestration configuration file defined based on the containerized application cluster management platform, the created data set information is obtained.

[0349] The step of performing storage arrangement on the data set according to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform to obtain a set of schedulable computing nodes capable of storing the data set includes:

[0350] According to the storage end specified in the data set information, query the occupied space of the data set;

[0351] Selecting candidate computing nodes that meet the screening requirements in the containerized application cluster management platform;

[0352] According to the occupied space information of the data set and the storage space of the candidate computing node, the data set is stored and arranged to obtain a set of schedulable computing nodes that can store the data set.

[0353] The step of selecting a candidate computing node that meets the screening requirements in the containerized application cluster management platform includes at least one of the following selection methods:

[0354] In the containerized application cluster management platform, selecting the candidate computing node that matches the label information defined in the data set information according to the label information of the computing node;

[0355] Selecting the candidate computing node that meets the availability zone range requirement defined in the data set information in the containerized application cluster management platform;

[0356] The candidate computing node that meets the specified device type defined in the data set information is selected in the containerized application cluster management platform.

[0357] Among them, it also includes:

[0358] According to the space occupied by the data set or the usage requirement of the data set, the data set is segmented to obtain at least two data set blocks;

[0359] Recording the name of the data set block and the amount of the data set block corresponding to the name that needs to be allocated to the containerized application cluster into the created data set record object;

[0360] The step of performing storage arrangement on the data set according to the occupied space information of the data set and the storage space of the candidate computing node to obtain a set of schedulable computing nodes capable of storing the data set includes:

[0361] According to the occupied space of the data set blocks in the data set record object and the storage space of the candidate computing nodes, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set.

[0362] Among them, it also includes:

[0363] Creating a computing node list for recording candidate computing node information according to the selected candidate computing nodes;

[0364] The step of performing storage arrangement on the data set according to the space occupied by the data set in the data set record object and the storage space of the candidate computing node to obtain a set of schedulable computing nodes capable of storing the data set includes:

[0365] According to the space occupied by the data set in the data set record object and the storage space of the candidate computing node in the computing node list, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set.

[0366] The step of creating a computing node list for recording information of the candidate computing nodes according to the selected candidate computing nodes includes:

[0367] Sorting the candidate computing nodes in ascending or descending order according to the storage capacity information of the available storage space;

[0368] The computing node list is created according to the sorted candidate computing nodes.

[0369] The step of performing storage arrangement on the data set according to the space occupied by the data set in the data set record object and the storage space of the candidate computing node to obtain a set of schedulable computing nodes capable of storing the data set includes:

[0370] Determine whether the computing node list has a set of available candidate computing nodes that meets a set availability threshold requirement;

[0371] If so, determining whether the space occupied by the data set is less than the storage capacity of the storage space of the available candidate computing nodes in the set of available candidate computing nodes;

[0372] If so, selecting an available candidate computing node that matches the occupancy from the set of available candidate computing nodes;

[0373] The available candidate computing nodes are determined as the set of schedulable computing nodes that can store the data set.

[0374] Among them, it also includes:

[0375] When the determination of whether the occupied amount of the space occupied by the data set is less than the storage amount of the storage space of the available candidate computing nodes in the set of available candidate computing nodes is no, then determining whether the occupied amount of the space occupied by the data set is greater than the storage amount of the storage space of the available candidate computing nodes in the set of available candidate computing nodes;

[0376] If yes, selecting the available candidate computing node with the largest storage capacity from the range of available candidate computing nodes;

[0377] The amount of space occupied by the data set is updated, and the process returns to the step of determining whether the computing node list has a set of available candidate computing nodes that meets the set availability threshold requirement;

[0378] The step of determining the available candidate computing nodes as the set of schedulable computing nodes capable of storing the data set comprises:

[0379] The selected available candidate computing node with the largest storage capacity is determined as the set of schedulable computing nodes that can store the data set.

[0380] Among them, it also includes:

[0381] When determining whether the occupied space of the data set is greater than the storage capacity of the storage space of the available candidate computing nodes in the set of available candidate computing nodes, if the result is no, selecting an available candidate computing node with the storage capacity equal to the occupied space from the range of the available candidate computing nodes;

[0382] The step of determining the available candidate computing nodes as the set of schedulable computing nodes capable of storing the data set comprises:

[0383] The selected available candidate computing nodes whose storage capacity is equal to the occupied capacity are determined as the schedulable computing node set.

[0384] The step of determining the available candidate computing nodes as the set of schedulable computing nodes capable of storing the data set includes:

[0385] The available candidate computing nodes are recorded in a scheduling list, where the scheduling list is used to record information about the set of schedulable computing nodes that can store the data set.

[0386] Among them, it also includes:

[0387] When the determination of whether there is a set of available candidate computing nodes that meet the set availability threshold requirements in the computing node list is no, exit the step of performing storage arrangement for the data set according to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform to obtain a set of schedulable computing nodes that can store the data set, or re-execute the step of selecting candidate computing nodes that meet the screening requirements in the containerized application cluster management platform, or wait according to the set waiting time.

[0388] Among them, it also includes:

[0389] When performing storage arrangement for the data set according to the space occupied by the data set in the data set record object and the storage space of the candidate computing node, determining whether the space occupied by the data set on the storage terminal changes;

[0390] If so, the data record object for recording the space occupied by the data set is recreated according to the data set after the storage amount is changed.

[0391] Among them, it also includes:

[0392] Acquire at least one of storage capacity information of the data set in the set of schedulable computing nodes and access capacity information of data in the data set stored in the set of schedulable computing nodes;

[0393] The information is used as reference information for application scheduling in the containerized application cluster management platform.

[0394] Alternatively, when the program is read and executed by the processor, the following operations are performed:

[0395] Based on the containerized application cluster management platform, determine the data set information that the application service needs to use;

[0396] Based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and orchestrated to obtain a set of schedulable computing nodes capable of storing the data set, and the scheduling information of the application service is determined;

[0397] According to the scheduling information, the application service is scheduled to a matching target computing node in the set of schedulable computing nodes.

[0398] The method of determining the data set information required for the application service based on the containerized application cluster management platform includes:

[0399] Based on the containerized application cluster management platform, the data set information to be used is specified in a configuration file for creating the application service.

[0400] Wherein, based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and arranged, and a set of schedulable computing nodes capable of storing the data set is obtained, and the scheduling information of the application service is determined, including:

[0401] Based on the set of schedulable computing nodes, configure selection configuration information that meets the node selection condition and path configuration information for accessing the computing nodes that meet the node selection condition;

[0402] The configuration file including the selection configuration information and the path configuration information is determined as the scheduling information of the application service.

[0403] The configuring, based on the set of schedulable computing nodes, the selection configuration information satisfying the node selection condition and the path configuration information for accessing the computing nodes satisfying the node selection condition includes:

[0404] Based on the created configuration file of the application service, the selection configuration information of the configuration satisfying the node selection condition is added, and the path configuration information of the access to the computing node satisfying the node selection condition is added.

[0405] The step of scheduling the application service to a matching target computing node in the set of schedulable computing nodes according to the scheduling information includes:

[0406] Determine the computing node that meets the node selection condition in the scheduling information as the target computing node;

[0407] The application service is scheduled according to the access path of the target computing node in the scheduling information.

[0408] Among them, it also includes:

[0409] Sorting the determined target computing nodes according to the storage capacity of storing the data set;

[0410] The step of scheduling the application service according to the access path of the target computing node in the scheduling information includes:

[0411] The application service is scheduled according to the access path of the target computing node with the largest storage capacity within the sorted target computing node range.

[0412] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0413] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0414] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0415] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0416] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A data processing method based on a containerized application cluster management platform, characterized in that: include: Obtain the dataset information created based on the containerized application cluster management platform; According to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and arranged to obtain a set of schedulable computing nodes that can store the data set; wherein, according to the occupied space of the data set or the use requirement of the data set, the data set is divided to obtain at least two data set blocks; the name of the data set block and the occupied amount of the data set block in the containerized application cluster corresponding to the name are allocated and recorded in the created data set record object; candidate computing nodes that meet the screening requirements are selected in the containerized application cluster management platform; according to the occupied space of the data set block in the data set record object and the storage space of the candidate computing nodes, the data set is stored and arranged; The data in the data set is preloaded into the set of schedulable computing nodes, on a target computing node of the storage space that matches the occupied space of the data.

2. The data processing method based on the containerized application cluster management platform according to claim 1 is characterized in that: The obtaining of data set information created based on the containerized application cluster management platform includes: According to the application orchestration configuration file defined based on the containerized application cluster management platform, the created data set information is obtained.

3. The data processing method based on the containerized application cluster management platform according to claim 1 is characterized in that: The step of performing storage arrangement on the data set according to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform to obtain a set of schedulable computing nodes capable of storing the data set includes: According to the storage end specified in the data set information, query the occupied space of the data set; Selecting candidate computing nodes that meet the screening requirements in the containerized application cluster management platform; According to the occupied space information of the data set and the storage space of the candidate computing node, the data set is stored and arranged to obtain a set of schedulable computing nodes that can store the data set.

4. The data processing method based on the containerized application cluster management platform according to claim 3 is characterized in that: The selecting of candidate computing nodes that meet the screening requirements in the containerized application cluster management platform includes at least one of the following selection methods: In the containerized application cluster management platform, selecting the candidate computing node that matches the label information defined in the data set information according to the label information of the computing node; Selecting the candidate computing node that meets the availability zone range requirement defined in the data set information in the containerized application cluster management platform; The candidate computing node that meets the specified device type defined in the data set information is selected in the containerized application cluster management platform.

5. The data processing method based on the containerized application cluster management platform according to claim 3 is characterized in that: Also includes: Creating a computing node list for recording candidate computing node information according to the selected candidate computing nodes; The step of performing storage arrangement on the data set according to the space occupied by the data set in the data set record object and the storage space of the candidate computing node to obtain a set of schedulable computing nodes capable of storing the data set includes: According to the space occupied by the data set in the data set record object and the storage space of the candidate computing node in the computing node list, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set.

6. The data processing method based on the containerized application cluster management platform according to claim 5 is characterized in that: The step of creating a computing node list for recording information of the candidate computing nodes according to the selected candidate computing nodes includes: Sorting the candidate computing nodes in ascending or descending order according to the storage capacity information of the available storage space; The computing node list is created according to the sorted candidate computing nodes.

7. The data processing method based on the containerized application cluster management platform according to claim 6 is characterized in that: The step of performing storage arrangement on the data set according to the space occupied by the data set in the data set record object and the storage space of the candidate computing node to obtain a set of schedulable computing nodes capable of storing the data set includes: Determine whether the computing node list has a set of available candidate computing nodes that meets a set availability threshold requirement; If so, determining whether the space occupied by the data set is less than the storage capacity of the storage space of the available candidate computing nodes in the set of available candidate computing nodes; If so, selecting an available candidate computing node that matches the occupancy from the set of available candidate computing nodes; The available candidate computing nodes are determined as the set of schedulable computing nodes that can store the data set.

8. The data processing method based on the containerized application cluster management platform according to claim 7 is characterized in that: Also includes: When the determination of whether the occupied amount of the space occupied by the data set is less than the storage amount of the storage space of the available candidate computing nodes in the set of available candidate computing nodes is no, then determining whether the occupied amount of the space occupied by the data set is greater than the storage amount of the storage space of the available candidate computing nodes in the set of available candidate computing nodes; If yes, selecting the available candidate computing node with the largest storage capacity from the range of available candidate computing nodes; The amount of space occupied by the data set is updated, and the process returns to the step of determining whether the computing node list has a set of available candidate computing nodes that meets the set availability threshold requirement; The step of determining the available candidate computing nodes as the set of schedulable computing nodes capable of storing the data set comprises: The selected available candidate computing node with the largest storage capacity is determined as the set of schedulable computing nodes that can store the data set.

9. The data processing method based on the containerized application cluster management platform according to claim 8, characterized in that: Also includes: When determining whether the occupied space of the data set is greater than the storage capacity of the storage space of the available candidate computing nodes in the set of available candidate computing nodes, if the result is no, selecting an available candidate computing node with the storage capacity equal to the occupied space from the range of the available candidate computing nodes; The step of determining the available candidate computing nodes as the set of schedulable computing nodes capable of storing the data set comprises: The selected available candidate computing nodes whose storage capacity is equal to the occupied capacity are determined as the schedulable computing node set.

10. The data processing method based on the containerized application cluster management platform according to claim 7, characterized in that: The step of determining the available candidate computing nodes as the set of schedulable computing nodes capable of storing the data set comprises: The available candidate computing nodes are recorded in a scheduling list, where the scheduling list is used to record information about the set of schedulable computing nodes that can store the data set.

11. The data processing method based on the containerized application cluster management platform according to claim 7, characterized in that: Also includes: When the determination of whether there is a set of available candidate computing nodes that meet the set availability threshold requirements in the computing node list is no, exit the step of performing storage arrangement for the data set according to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform to obtain a set of schedulable computing nodes that can store the data set, or re-execute the step of selecting candidate computing nodes that meet the screening requirements in the containerized application cluster management platform, or wait according to the set waiting time.

12. The data processing method based on the containerized application cluster management platform according to claim 3, characterized in that: Also includes: When performing storage arrangement for the data set according to the space occupied by the data set in the data set record object and the storage space of the candidate computing node, determining whether the space occupied by the data set on the storage terminal changes; If so, a data record object for recording the space occupied by the data set is recreated according to the data set after the storage capacity is changed.

13. The data processing method based on the containerized application cluster management platform according to claim 1, characterized in that: Also includes: Acquire at least one of storage capacity information of the data set in the set of schedulable computing nodes and access capacity information of data in the data set stored in the set of schedulable computing nodes; The information is used as reference information for application scheduling in the containerized application cluster management platform.

14. A data processing device based on a containerized application cluster management platform, characterized in that: include: An acquisition unit, used to acquire data set information created based on a containerized application cluster management platform; An orchestration unit is used to perform storage orchestration on the data set according to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, and obtain a set of schedulable computing nodes that can store the data set; wherein, according to the occupied space of the data set or the use requirement of the data set, the data set is segmented to obtain at least two data set blocks; the name of the data set block and the occupied amount of the data set block in the containerized application cluster corresponding to the name are allocated and recorded in the created data set record object; candidate computing nodes that meet the screening requirements are selected in the containerized application cluster management platform; storage orchestration on the data set is performed according to the occupied space of the data set block in the data set record object and the storage space of the candidate computing nodes; A preloading unit is used to preload the data in the data set into the schedulable computing node, on a target computing node of the storage space that matches the occupied space of the data.

15. An application scheduling method based on a containerized application cluster management platform, characterized in that: include: Based on the containerized application cluster management platform, determine the data set information that the application service needs to use; Based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and orchestrated, and a set of schedulable computing nodes that can store the data set is obtained, and the scheduling information of the application service is determined; wherein, according to the occupied space of the data set or the usage requirements of the data set, the data set is segmented to obtain at least two data set blocks; the name of the data set block and the occupied amount of the data set block corresponding to the name that needs to be allocated to the containerized application cluster are recorded in the created data set record object; candidate computing nodes that meet the screening requirements are selected in the containerized application cluster management platform; according to the occupied space of the data set block in the data set record object and the storage space of the candidate computing node, the data set is stored and orchestrated; According to the scheduling information, the application service is scheduled to a matching target computing node in the set of schedulable computing nodes.

16. The application scheduling method based on the containerized application cluster management platform according to claim 15, characterized in that: The containerized application cluster management platform is used to determine the data set information that the application service needs to use, including: Based on the containerized application cluster management platform, the data set information to be used is specified in a configuration file for creating the application service.

17. The application scheduling method based on the containerized application cluster management platform according to claim 15, characterized in that: The step of performing storage arrangement on the data set based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform to obtain a set of schedulable computing nodes capable of storing the data set and determining the scheduling information of the application service includes: Based on the set of schedulable computing nodes, configure selection configuration information that meets the node selection condition and path configuration information for accessing the computing nodes that meet the node selection condition; The configuration file including the selection configuration information and the path configuration information is determined as the scheduling information of the application service.

18. The application scheduling method based on the containerized application cluster management platform according to claim 17, characterized in that: The configuring, based on the set of schedulable computing nodes, selection configuration information satisfying a node selection condition and path configuration information for accessing the computing nodes satisfying the node selection condition comprises: Based on the created configuration file of the application service, the selection configuration information of the configuration satisfying the node selection condition is added, and the path configuration information of the access to the computing node satisfying the node selection condition is added.

19. The application scheduling method based on the containerized application cluster management platform according to claim 17, characterized in that: Scheduling the application service to a matching target computing node in the set of schedulable computing nodes according to the scheduling information includes: Determine the computing node that meets the node selection condition in the scheduling information as the target computing node; The application service is scheduled according to the access path of the target computing node in the scheduling information.

20. The application scheduling method based on the containerized application cluster management platform according to claim 19, characterized in that: Also includes: Sorting the determined target computing nodes according to the storage capacity of storing the data set; The scheduling of the application service according to the access path of the target computing node in the scheduling information includes: The application service is scheduled according to the access path of the target computing node with the largest storage capacity within the sorted target computing node range.

21. An application scheduling device based on a containerized application cluster management platform, characterized in that: include: A first determination unit is used to determine data set information that the application service needs to use based on the containerized application cluster management platform; A second determination unit is used to determine the scheduling information of the application service by performing storage orchestration on the data set based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, and obtaining a set of schedulable computing nodes that can store the data set; wherein, according to the occupied space of the data set or the use requirement of the data set, the data set is segmented to obtain at least two data set blocks; the name of the data set block and the occupied amount of the data set block in the containerized application cluster corresponding to the name are allocated and recorded in the created data set record object; candidate computing nodes that meet the screening requirements are selected in the containerized application cluster management platform; storage orchestration is performed on the data set according to the occupied space of the data set block in the data set record object and the storage space of the candidate computing node; The scheduling unit is used to schedule the application service to a matching target computing node in the set of schedulable computing nodes according to the scheduling information.

22. A computer storage medium for storing data generated by a network platform, and a program for processing data generated by the network platform; When the program is read and executed, the following operations are performed: Obtain the dataset information created based on the containerized application cluster management platform; According to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and arranged to obtain a set of schedulable computing nodes capable of storing the data set; wherein, According to the space occupied by the data set or the usage requirement of the data set, the data set is segmented to obtain at least two data set blocks; Recording the name of the data set block and the amount of the data set block corresponding to the name that needs to be allocated to the containerized application cluster into the created data set record object; Selecting candidate computing nodes that meet the screening requirements in the containerized application cluster management platform; Performing storage arrangement for the data set according to the occupied space of the data set blocks in the data set record object and the storage space of the candidate computing node; Preloading the data in the data set into the set of schedulable computing nodes, onto a target computing node of the storage space that matches the occupied space of the data; or, Based on the containerized application cluster management platform, determine the data set information that the application service needs to use; Based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and orchestrated, and a set of schedulable computing nodes that can store the data set is obtained, and the scheduling information of the application service is determined; wherein, according to the occupied space of the data set or the usage requirements of the data set, the data set is segmented to obtain at least two data set blocks; the name of the data set block and the occupied amount of the data set block corresponding to the name that needs to be allocated to the containerized application cluster are recorded in the created data set record object; candidate computing nodes that meet the screening requirements are selected in the containerized application cluster management platform; according to the occupied space of the data set block in the data set record object and the storage space of the candidate computing node, the data set is stored and orchestrated; According to the scheduling information, the application service is scheduled to a matching target computing node in the set of schedulable computing nodes.

23. An electronic device, comprising: processor; The memory is used to store a program for processing data generated by the network platform, and when the program is read and executed by the processor, the program performs the following operations: Obtain the dataset information created based on the containerized application cluster management platform; According to the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and arranged to obtain a set of schedulable computing nodes that can store the data set; wherein, according to the occupied space of the data set or the use requirement of the data set, the data set is divided to obtain at least two data set blocks; the name of the data set block and the occupied amount of the data set block in the containerized application cluster corresponding to the name are allocated and recorded in the created data set record object; candidate computing nodes that meet the screening requirements are selected in the containerized application cluster management platform; according to the occupied space of the data set block in the data set record object and the storage space of the candidate computing nodes, the data set is stored and arranged; Preloading the data in the data set into the set of schedulable computing nodes, onto a target computing node of the storage space that matches the occupied space of the data; or, Based on the containerized application cluster management platform, determine the data set information that the application service needs to use; Based on the occupied space of the data set in the data set information and the storage space of the computing nodes in the containerized application cluster management platform, the data set is stored and orchestrated, and a set of schedulable computing nodes that can store the data set is obtained, and the scheduling information of the application service is determined; wherein, according to the occupied space of the data set or the usage requirements of the data set, the data set is segmented to obtain at least two data set blocks; the name of the data set block and the occupied amount of the data set block corresponding to the name that needs to be allocated to the containerized application cluster are recorded in the created data set record object; candidate computing nodes that meet the screening requirements are selected in the containerized application cluster management platform; according to the occupied space of the data set block in the data set record object and the storage space of the candidate computing node, the data set is stored and orchestrated; According to the scheduling information, the application service is scheduled to a matching target computing node in the set of schedulable computing nodes.

Images