Timing capacity expansion method and device, computer equipment and readable storage medium

Abstract

The invention relates to a timed capacity expansion method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a self-defined resource configuration file; the resource configuration file comprises self-defined capacity expansion time, a capacity expansion target and a capacity expansion strategy; generating a timed capacity expansion task of the preset distributed management platform according to the resource configuration file; regularly expanding the native expansion task resources of the distributed management platform; and when the distributed management platform determines that the capacity expansion time is up, determining a capacity expansion strategy and a capacity expansion target based on the timed capacity expansion task, and performing capacity expansion processing on the capacity expansion target according to the capacity expansion strategy. By adopting the method, timed capacity expansion can be realized.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a timed expansion method, apparatus, computer device, and readable storage medium. Background Technology

[0002] Currently, big data computing frameworks such as Flink and Spark primarily run on Kubernetes clusters. Kubernetes (K8s) is an open-source distributed management platform primarily used for automating the deployment, scaling, and management of containerized applications.

[0003] Resource management for each node in the cluster primarily relies on native mechanisms provided by Kubernetes. For example, Kubernetes scales up nodes based on their CPU or memory usage to adapt to changes in workload. However, this is not suitable for scheduled tasks (such as peak traffic periods at specific times of the day). Summary of the Invention

[0004] Therefore, it is necessary to provide a timed capacity expansion method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can perform timed capacity expansion in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a timed capacity expansion method, including:

[0006] Obtain a custom resource configuration file; the resource configuration file includes a custom expansion time, expansion target, and expansion strategy;

[0007] Based on the resource configuration file, a scheduled expansion task for the preset distributed management platform is generated; the scheduled expansion task is the native expansion task resource of the distributed management platform.

[0008] When the distributed management platform determines that the expansion time has arrived, it determines the expansion strategy and expansion target based on the scheduled expansion task, and performs expansion processing on the expansion target according to the expansion strategy.

[0009] In one embodiment, the method further includes:

[0010] Determine the preset distributed management platform and obtain the custom resource definition of the distributed management platform;

[0011] Determine the expansion timeline, expansion targets, and expansion strategies;

[0012] A custom resource configuration file is generated based on the format specifications defined in the custom resource definition, the expansion time, the expansion target, and the expansion strategy.

[0013] In one embodiment, generating a scheduled scaling task for a preset distributed management platform based on the resource configuration file includes:

[0014] Extract key configuration information from the resource configuration file;

[0015] Based on the aforementioned key configuration information, generate a task configuration file for the scheduled expansion task;

[0016] The task configuration file is submitted to the distributed management platform so that the distributed management platform can generate a scheduled expansion task based on the task configuration file.

[0017] In one embodiment, the key configuration information includes expansion time, expansion target, resource configuration file name, and expansion strategy name;

[0018] The step of generating a task configuration file for the scheduled expansion task based on the configured key information includes:

[0019] The expansion time extracted from the resource configuration file will be filled into the expansion time field in the task configuration file template.

[0020] Fill the container startup command field in the task configuration file template with the expansion target, the configuration file name, and the expansion strategy name to obtain the task configuration file.

[0021] In one embodiment, when the distributed management platform determines that the expansion time has arrived, determining the expansion strategy and expansion target based on the scheduled expansion task includes:

[0022] When the expansion time recorded in the scheduled expansion task is reached, the expansion target, resource configuration file name, and expansion strategy name recorded in the expansion task are determined.

[0023] The resource configuration file corresponding to the resource configuration file name is read, and the expansion strategy corresponding to the expansion strategy name is read from the read resource configuration file.

[0024] In one embodiment, scaling up the target node in the distributed cluster corresponding to the scaling target according to the scaling strategy includes:

[0025] Determine the operating platform corresponding to the expansion target indicated by the timed expansion task;

[0026] According to the hardware resource expansion strategy in the expansion strategy, the hardware resources of the running carrier are expanded;

[0027] According to the parallel expansion strategy in the expansion strategy, the parallelism of the running carrier is expanded.

[0028] In one embodiment, expanding the hardware resources of the running platform according to the hardware resource expansion strategy in the expansion strategy includes:

[0029] Delete the aforementioned operating platform;

[0030] According to the hardware resource expansion strategy in the expansion strategy, a new running platform is created so that the running platform has the hardware resources corresponding to the hardware resource expansion strategy.

[0031] In one embodiment, creating a new runtime environment according to the hardware resource expansion strategy in the expansion strategy includes:

[0032] Based on the hardware resource expansion strategy in the expansion strategy, the configuration information of the cluster deployment object of the distributed management platform is modified;

[0033] When the native controller of the distributed management platform detects a change in the configuration information of the cluster deployment object, it creates a new runtime environment based on the configuration information of the cluster deployment object.

[0034] In one embodiment, expanding the parallelism of the runtime carrier according to the parallel expansion strategy in the expansion strategy includes:

[0035] According to the parallel expansion strategy in the expansion strategy, the operator parallelism of the new running platform is adjusted while keeping the new running platform running.

[0036] In one embodiment, the method further includes:

[0037] Before scaling up, store the original specifications corresponding to the scaling target.

[0038] Once the expansion is complete, store the expanded specifications corresponding to the expansion target, and store the expansion execution status.

[0039] In one embodiment, the method further includes:

[0040] When the expansion window ends, the expanded specifications will be rolled back to the original specifications for the expansion target.

[0041] Secondly, this application also provides a timed capacity expansion device, comprising:

[0042] The file acquisition module is used to acquire a custom resource configuration file; the resource configuration file includes a custom expansion time, expansion target, and expansion strategy;

[0043] The task generation module is used to generate a scheduled expansion task for a preset distributed management platform based on the resource configuration file; the scheduled expansion task is a native expansion task resource of the distributed management platform.

[0044] The expansion processing module is used to determine the expansion strategy and expansion target based on the timed expansion task when the distributed management platform determines that the expansion time has arrived, and to perform expansion processing on the expansion target according to the expansion strategy.

[0045] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in any of the timed expansion methods provided in the embodiments of this application.

[0046] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the timed expansion methods provided in the embodiments of this application.

[0047] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in any of the timed expansion methods provided in the embodiments of this application.

[0048] The aforementioned timed expansion method, apparatus, computer equipment, computer-readable storage medium, and computer program product, by acquiring a resource configuration file, can convert the resource configuration file into a timed expansion task that the distributed management platform can understand. Then, when the expansion time indicated by the timed expansion task arrives, expansion can be performed on the expansion target indicated by the timed expansion task according to the expansion strategy specified in the task. In traditional solutions, the distributed management platform mainly expands based on the CPU or memory usage of nodes in the cluster, and cannot achieve timed expansion. In this application, since a custom resource configuration file specifying the expansion time can be converted into a timed expansion task, and timed expansion is a native expansion task resource of the distributed management platform, the distributed management platform can understand the timed expansion task and thus perform timed expansion processing based on the timed expansion task. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a diagram illustrating the application environment of a timed capacity expansion method in one embodiment.

[0051] Figure 2 This is a flowchart illustrating a timed capacity expansion method in one embodiment;

[0052] Figure 3 This is a schematic diagram of a timed expansion architecture in one embodiment;

[0053] Figure 4 This is an interactive diagram illustrating timed capacity expansion in one embodiment;

[0054] Figure 5 This is a structural block diagram of a timed expansion device in one embodiment;

[0055] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0058] The timed capacity expansion method provided in this application embodiment can be applied to, for example... Figure 1The application environment is shown. Terminal 102 communicates with cluster 104 via a network. A data storage system can store the data that cluster 104 needs to process. The data storage system can be integrated into cluster 104 or placed in the cloud or on other network servers. Users can edit resource configuration files on terminal 102, which can then send the resource configuration files to the worker nodes in cluster 104. The worker nodes can then perform periodic expansion processing according to the resource configuration files. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Cluster 104 includes multiple nodes, which can be independent physical servers or cloud servers.

[0059] In one exemplary embodiment, such as Figure 2 As shown, a timed expansion method is provided, which can be applied to... Figure 1 Taking cluster 104 as an example, the explanation includes steps 202 to 206. Wherein:

[0060] Step 202: Obtain the custom resource configuration file.

[0061] The resource configuration file (also known as CronScaler CR) is a configuration file generated based on custom resource definitions. It can contain configuration scaling targets (such as the Flink job name and its Kubernetes namespace), scaling times (such as triggering every Friday at 17:00), and scaling strategies (including details such as CPU / memory hardware resource adjustments and operator parallelism optimization). Flink is a big data computing framework.

[0062] Custom resource definitions (CRs) are extension interfaces provided by distributed management platforms. CRs predefine the required fields for resource configuration files, enabling the distributed management platform to recognize resource configuration files generated based on them. Specifically, the distributed management platform can be Kubernetes.

[0063] Kubernetes (K8s) is an open-source distributed management platform that is primarily used for automating the deployment, scaling, and management of containerized applications.

[0064] Optionally, a timed scaling controller (also known as a Cronscaler-Controller) is deployed on the compute nodes in the cluster. The timed scaling controller can listen to the resource configuration file through the interface provided by the distributed management platform. When a user submits or updates the resource configuration file, the timed scaling controller will detect it in real time and pull the resource configuration file.

[0065] Step 204: Generate a scheduled expansion task for the preset distributed management platform based on the resource configuration file; the scheduled expansion task is a native expansion task resource of the distributed management platform.

[0066] Among them, the scheduled scaling task (also known as CronJob) is a native scaling task resource of the distributed management platform. It is a task that is periodically triggered according to the scaling expression of the distributed management platform (such as "0 17 * * FRI", which means every Friday at 5 PM). Since the scheduled scaling task is a native scaling task resource of the distributed management platform, it can be recognized and understood by the distributed management platform.

[0067] Optionally, when obtaining the resource configuration file, the scheduled scaling controller can generate scheduled scaling tasks for the distributed management platform (e.g., Kubernetes) based on the content recorded in the resource configuration file. For example, the scheduled scaling controller generates a task configuration file for the scheduled scaling task based on the content recorded in the resource configuration file, so that the distributed management platform can generate the scheduled scaling task based on the task configuration file.

[0068] Step 206: When the distributed management platform determines that the expansion time has arrived, it determines the expansion strategy and expansion target based on the scheduled expansion task, and performs expansion processing on the expansion target according to the expansion strategy.

[0069] Among them, the expansion strategy refers to the specific execution plan for expansion and contraction. For example, the expansion strategy includes the hardware resource adjustment strategy, the operator instance number adjustment strategy, the execution order between the hardware resource adjustment strategy and the operator instance number adjustment strategy, and so on.

[0070] The scaling target is the business job within a specified namespace in the distributed cluster. Business jobs can be, for example, Flink jobs such as "real-time statistics of e-commerce order amounts," "offline cleaning of user behavior logs," and "monitoring of abnormal data streams in the system."

[0071] A distributed cluster, such as a Kubernetes cluster that runs business jobs.

[0072] Optionally, when the scheduled scaling time indicated by the timed scaling task arrives, the distributed management platform creates a temporary running container and runs the timed scaling task executor (also known as cronscaler-executor) in the temporary running container to scale according to the scaling strategy and scaling target indicated by the timed scaling task.

[0073] Optionally, the timed expansion task records the expansion time, for example, "0 17 * * FRI". When the expansion time is reached, the expansion strategy and expansion target indicated by the timed expansion task are determined.

[0074] Optionally, the scheduled expansion task records the job name and namespace. The expansion target can be determined based on the job name and namespace. This expansion target is the expansion target indicated by the scheduled expansion task.

[0075] Optionally, the scheduled expansion task records the file name of the resource configuration file. The corresponding resource configuration file can be determined based on the resource configuration file name, and the expansion strategy can be read from the resource configuration file. This expansion strategy is the expansion strategy indicated by the scheduled expansion task.

[0076] Optionally, after the scheduled scaling task is triggered, the scaling executor in the compute node can perform a complete scaling operation on the corresponding runtime environment for the precisely located scaling target within the distributed cluster, according to the scaling strategy. This could include scaling up hardware resources and parallel operators. The runtime environment refers to the container that supports the scaling target's operation. For example, the runtime environment could be the Pod corresponding to the JobManager (task management node) and TaskManager (compute node) in the cluster, which is the physical container that carries CPU, memory resources, and job execution. A pod is the smallest deployable unit in Kubernetes, used to manage a group of containers sharing storage, network, and runtime environment.

[0077] Optionally, if the expansion strategy includes hardware resource expansion, the number of CPU cores, memory specifications, etc. of the running vehicle running the expansion target will be adjusted; if the expansion strategy includes parallelism expansion, the operator parallelism of the running vehicle will be adjusted.

[0078] In the aforementioned timed expansion method, by obtaining the resource configuration file, the resource configuration file can be converted into a timed expansion task that the distributed management platform can understand. Then, when the expansion time indicated by the timed expansion task arrives, the expansion target indicated by the timed expansion task can be expanded according to the expansion strategy indicated by the timed expansion task. In traditional solutions, the distributed management platform mainly expands based on the CPU or memory usage of nodes in the cluster, and cannot achieve timed expansion. In this application, since a custom resource configuration file specifying the expansion time can be converted into a timed expansion task, and timed expansion is a native expansion task resource of the distributed management platform, the distributed management platform can understand the timed expansion task and thus perform timed expansion processing based on the timed expansion task.

[0079] In one embodiment, the method further includes: determining a preset distributed management platform and obtaining a custom resource definition of the distributed management platform; determining the expansion time, expansion target, and expansion strategy; and generating a custom resource configuration file based on the format specification, expansion time, expansion target, and expansion strategy defined in the custom resource definition.

[0080] Optionally, a distributed management platform can be identified, and its custom resource definition (CRD) can be obtained. For example, if the distributed management platform is identified as Kubernetes, its custom resource definition (CRD) can be obtained. Further, the custom scaling time, scaling target, and scaling strategy can be obtained. Based on the format specifications, scaling time, scaling target, and scaling strategy defined in the custom resource definition, a custom resource configuration file is generated. The custom resource configuration file conforms to the format specifications defined in the custom resource definition. For example, the generated resource configuration file may include the following: resource configuration file name, target framework (the big data component to be scaled, such as Flink), scaling target (such as the name of the Flink job to be scaled), namespace where the scaling target is located, scaling strategy name, and scaling time (the scaling strategy written using a Cron expression, such as cron: "0 17 * *FRI").

[0081] The resource configuration file can also record scaling strategies, which can be as follows: Hardware resource adjustment strategy (requires job restart): For task management nodes, after scaling, the number of CPU cores (2 cores) and memory (4096Mi=4GB) will be increased; for compute nodes, the number of CPU cores (4 cores) and memory (8192Mi=8GB) and number of slots (10 slots, determining the number of runnable subtasks) will be increased; Parallelism adjustment strategy (no job restart required): For data source operators, after scaling, the parallelism will be increased (4 instances read data simultaneously); For business processing operators (such as data filtering and aggregation), after scaling, the parallelism will be increased (8 instances process data simultaneously).

[0082] In this embodiment, by defining a custom resource definition, a resource configuration file can be written according to the format specification defined by the custom resource definition, so that the resource configuration file can clearly define the expansion time, expansion target and expansion strategy.

[0083] In one embodiment, generating a scheduled expansion task for a pre-defined distributed management platform based on a resource configuration file includes: parsing configuration key information from the resource configuration file; generating a task configuration file for the scheduled expansion task based on the configuration key information; and submitting the task configuration file to the distributed management platform so that the distributed management platform generates the scheduled expansion task based on the task configuration file.

[0084] Optionally, upon obtaining the resource configuration file, the scheduled scaling controller can parse key configuration information from it, such as the scaling time and resource configuration file name. Following the native resource specifications of the distributed management platform, this information is then populated into a pre-defined task configuration file template to generate a complete task configuration file. This file clearly defines the task's trigger time, execution logic, resource requirements, and restart strategy, ensuring the distributed management platform can directly recognize and parse it. Finally, the scheduled scaling controller submits the task configuration file to the distributed management platform via a pre-defined interface (e.g., the Kubernetes APIServer). After verifying the configuration's validity, the distributed management platform automatically creates the corresponding scheduled scaling task (CronJob resource) and manages it through its own scheduling system, waiting for the pre-defined time to trigger subsequent scaling operations.

[0085] In one embodiment, the configuration key information includes expansion time, expansion target, resource configuration file name, and expansion strategy name; based on the configuration key information, a task configuration file for the scheduled expansion task is generated, including: filling the expansion time extracted from the resource configuration file into the expansion time field in the task configuration file template; and filling the expansion target, configuration file name, and expansion strategy name into the container startup command field in the task configuration file template to obtain the task configuration file.

[0086] Optionally, a task configuration file template can be obtained, and the expansion time extracted from the resource configuration file can be filled into the expansion time field in the task configuration file template. The expansion target, configuration file name, and expansion strategy name can be filled into the container startup command field in the task configuration file template to obtain the task configuration file.

[0087] Optionally, the generated task configuration file may include: `schedule: "0 17 * * FRI"` and `command: ["cronscaler-executor", "--jobName=flink-job", "--namespace=flink-test", "--cronscaler_name=example-cronscaler", "--schedule_name=friday-scaling"]`. Here, `schedule: "0 17 * * FRI"` indicates the scaling time (triggered every Friday at 5 PM). `command` represents the container startup command. `--jobName=flink-job` represents the name of the scaling target. `--namespace=flink-test` represents the namespace. `--cronscaler_name=example-cronscaler` represents the name of the associated resource configuration file. `--schedule_name=friday-scaling` represents the name of the scaling strategy to be executed. The scheduled scaling task generated based on this task configuration file indicates that the scaling task will be triggered every Friday at 5 PM, and the scheduled task executor will be invoked to perform the actual scaling operation.

[0088] In the above embodiments, resource configuration files that the distributed management platform cannot understand are converted into scheduled expansion tasks. Since the scheduled expansion task is a native expansion task resource of the distributed management platform, the distributed management platform can understand the scheduled expansion task and thus use the scheduling capability of the distributed management platform to ensure the execution of the scheduled expansion task, ultimately achieving scheduled expansion.

[0089] In one embodiment, when the distributed management platform determines that the expansion time has arrived, it determines the expansion strategy and expansion target based on the scheduled expansion task, including: when the expansion time recorded in the scheduled expansion task arrives, determining the expansion target, resource configuration file name, and expansion strategy name recorded in the expansion task; reading the corresponding resource configuration file through the resource configuration file name, and reading the expansion strategy corresponding to the expansion strategy name from the read resource configuration file.

[0090] Optionally, when the expansion time recorded in the scheduled expansion task arrives, the native controller of the distributed management platform (e.g., the CronJob controller) responds immediately, creates a temporary running platform according to the running platform job template recorded in the scheduled expansion task, and runs the scheduled task executor in the temporary running platform. The scheduled task executor determines the expansion strategy and expansion target, and expands the running platform corresponding to the expansion target according to the expansion strategy.

[0091] Optionally, the native controller of the distributed management platform will automatically create a short-lived runtime (i.e., a temporary runtime) based on the job template of the scheduled scaling task. This temporary runtime is only used to execute the current scaling operation. After the temporary runtime starts, the internal scheduled task executor will actively read the key parameters carried in the scheduled scaling task, including the name of the scaling target, the namespace where the scaling target is located, the name of the associated resource configuration file, and the name of the associated scaling strategy. Then, based on the name of the scaling target and the namespace where the scaling target is located, it will determine the scaling target, and based on the name of the resource configuration file and the name of the scaling strategy, it will determine the scaling strategy.

[0092] In one embodiment, according to the expansion strategy, the target node in the distributed cluster corresponding to the expansion target is expanded, including: determining the running vehicle corresponding to the expansion target indicated by the timed expansion task; expanding the hardware resources of the running vehicle according to the hardware resource expansion strategy in the expansion strategy; and expanding the parallelism of the running vehicle according to the parallel expansion strategy in the expansion strategy.

[0093] Optionally, after the scheduled expansion task is triggered, the runtime environment corresponding to the expansion target can be determined, that is, the core containerized unit supporting the operation of the expansion target, including the runtime environment corresponding to the computing nodes and the runtime environment corresponding to the management and control nodes. These runtime environments are the direct entities that carry out the operation of the expansion target and allocate hardware resources. Furthermore, according to the hardware resource expansion strategy in the expansion strategy, the hardware resources of the runtime environment can be expanded, and according to the parallel expansion strategy in the expansion strategy, the parallelism of the runtime environment can be expanded, thereby realizing multi-type expansion of software and hardware.

[0094] In one embodiment, the hardware resources of the running vehicle are expanded according to the hardware resource expansion strategy in the expansion strategy, including: deleting the running vehicle; and creating a new running vehicle according to the hardware resource expansion strategy in the expansion strategy, so that the running vehicle has the hardware resources corresponding to the hardware resource expansion strategy.

[0095] Optionally, when the timed scaling controller determines the hardware resource scaling strategy, it can modify the configuration information of the cluster deployment object according to the hardware resource scaling strategy through the interface provided by the distributed management platform. Then, when the deployment controller of the distributed management platform (e.g., Flink operator) detects that the configuration information of the cluster deployment object (e.g., FlinkDeployment) has been modified, it will delete the old runtime environment and create a new runtime environment according to the hardware resource scaling strategy, so that the runtime environment has the hardware resources corresponding to the hardware resource scaling strategy.

[0096] In one embodiment, creating a new runtime environment according to the hardware resource expansion strategy in the expansion strategy includes: modifying the configuration information of the cluster deployment object of the distributed management platform according to the hardware resource expansion strategy in the expansion strategy; and when the native controller of the distributed management platform detects a change in the configuration information of the cluster deployment object, creating a new runtime environment according to the configuration information of the cluster deployment object.

[0097] Optionally, when performing hardware resource expansion, the configuration information of the cluster deployment object of the distributed management platform can be modified according to the hardware resource expansion strategy in the expansion strategy. For example, the scheduled task executor calls the preset interface to modify the configuration (hardware resource part) of the cluster deployment object. For example, for the expansion target, the configuration of the computing node is as follows: after expansion, the CPU will have 2 cores and the memory will be 4GB; the configuration of the task management node is as follows: after expansion, the CPU will have 1 core, the memory will be 8GB, and the number of slots will be 10.

[0098] Furthermore, the deployment controller will first delete the old runtime environments (including the runtime environments corresponding to compute nodes and task management nodes) that are currently supporting the expansion target. After the old runtime environments have completely terminated, new runtime environments can be created based on the modified configuration information of the cluster deployment object. The modified configuration information specifies key parameters such as the number of CPU cores, memory capacity, and number of slots after expansion. Therefore, corresponding hardware resources can be allocated to the new runtime environments based on these configurations, ensuring that the new runtime environments have the resource specifications required by the hardware resource expansion strategy from the very beginning.

[0099] In the above embodiments, by deleting the old operating platform and creating a new operating platform, the hardware resource expansion strategy is ensured to be accurately implemented, so that the expansion target can obtain sufficient hardware resources and significantly improve the hardware support capability for data processing.

[0100] In one embodiment, the parallelism of the runtime is expanded according to the parallel expansion strategy in the expansion strategy, including: adjusting the operator parallelism of the new runtime while keeping the new runtime running, according to the parallel expansion strategy in the expansion strategy.

[0101] Optionally, after hardware resource expansion is completed and the new runtime is running stably, operator parallelism adjustment operations can be performed according to the parallelism expansion rules in the expansion strategy, without interrupting the new runtime. This process does not require restarting the new runtime or stopping the job. Instead, Flink's compute nodes send parallelism adjustment instructions to the new runtime to dynamically increase or optimize the number of instances of each operator in the job. For example, the operator parallelism in the data reading stage can be adjusted from 4 to 8, or the operator parallelism in the data aggregation stage can be increased from 6 to 12.

[0102] In the above implementation, with sufficient hardware resources on the new operating platform, the hardware resources and logical processing capabilities are precisely matched by dynamically adjusting the parallelism of operators. This not only eliminates the need to restart the operating platform or interrupt the job, ensuring the continuity of data processing and the stability of business, but also fully releases the performance potential of the expanded hardware resources, allowing the number of operator instances to match the hardware carrying capacity and significantly improving the concurrent processing efficiency of the job.

[0103] In one embodiment, when expanding capacity, the hardware resources of CPU / memory are expanded first, and the parallelism is adjusted after the running platform restarts; when shrinking capacity, the parallelism is adjusted first, and then the hardware resources are reclaimed.

[0104] In one embodiment, the method further includes: before scaling up, storing the pre-scaling specifications corresponding to the scaling target; after scaling up is completed, storing the post-scaling specifications corresponding to the scaling target, and storing the scaling up execution status.

[0105] Optionally, before scaling up, the pre-scaling specifications corresponding to the scaling target can be recorded in the resource configuration file. After scaling up, the post-scaling specifications corresponding to the scaling target can also be stored, along with the scaling execution status. For example, after scaling up is complete, the time of the most recent scaling up, the scaling up result, the resource rules after scaling up, and the resource specifications before scaling up can be recorded.

[0106] In one embodiment, the method further includes: when the expansion time window ends, rolling back the expanded specification to the original specification for the expansion target.

[0107] Optionally, when the expansion window ends, the expanded specifications can be rolled back to the original specifications for the expansion target. For example, the expanded specifications can be rolled back to the original specifications based on the original specifications recorded in the resource configuration file.

[0108] In one embodiment, if the expansion fails within a certain time period, a rollback can be performed by selecting one of the previous expansion specifications.

[0109] In the above embodiments, when the expansion time window ends, rolling back the resources can reduce cloud computing costs and improve resource utilization.

[0110] In one embodiment, reference Figure 3 Users submit resource configuration files (recording scaling time, strategy, scaling targets, etc.) and configuration information of cluster deployment objects (defining Flink cluster resources, job information, etc.) to the Kubernetes cluster through the Kubernetes API-Server (the cluster's API entry point). After the resource configuration files and configuration information are submitted, they are managed by the Kubernetes cluster. The scheduled scaling controller listens to the resource configuration files and triggers the scaling process according to the scheduled rules; the deployment controller listens to the cluster deployment objects and updates the runtime environment based on the resource information of the cluster deployment objects. Figure 3 A schematic diagram of a timed expansion architecture is shown in one embodiment.

[0111] In one embodiment, reference Figure 4 , Figure 4 This diagram illustrates the interaction of timed scaling in one embodiment. The timed scaling system includes an interaction portal, a timed scaling controller, and cluster nodes. Resource configuration files can be created through the interaction portal. The timed scaling controller reads the user-defined resource configuration file, including the scaling target (Flink job), scaling time (Cron expression), and scaling strategy (resource / parallelism adjustment), and then generates a timed scaling task based on the resource configuration file. Scaling operations can be executed according to the scaling time specified in the timed scaling task. CPU / memory scaling requires a restart, while parallelism scaling does not. By distinguishing between restartable and non-restartable scaling strategies, unnecessary job interruptions are avoided. The timed scaling controller can send requests to the cluster nodes, which then return the scaling results to the timed scaling controller, which can store the scaling results in the resource configuration file. When hardware resource scaling is performed, the fields in the resource configuration file are updated based on the scaling results after the hardware resource scaling is complete. When parallelism scaling is performed, the fields in the resource configuration file are updated based on the parallelism scaling results.

[0112] In the above embodiments, a preset expression (e.g., a Cron expression) is used to configure the scaling time, meeting the resource scheduling needs during peak business periods and improving task stability. By automatically distinguishing between restart-type and non-restart-type scaling, job interruptions are also reduced. Scaling is performed within a specified time period, and automatic rollback occurs after completion, avoiding long-term resource occupation and reducing cloud computing costs. The scaling solution of this application is scalable to different versions of big data frameworks, suitable for various big data computing environments, enhancing versatility. By recording the resource status before and after scaling, a scaling history is provided, facilitating debugging and maintenance, and improving the controllability and observability of scaling.

[0113] In summary, this application utilizes a distributed management platform's scheduled scaling task for timed scaling up and down, and combines platform operators (e.g., Flink-Kubernetes-Operator) and platform interfaces (e.g., Flink REST API) to achieve resource management. 1.1 This application employs a scheduled scaling task as the scheduling mechanism, automatically triggering scaling up and down tasks according to the scheduling rules (Cron expressions) configured by the custom resource definition. 1.2 The scheduled scaling controller of this application monitors changes in resource configuration files and dynamically generates scheduled scaling up tasks based on these files, running the tasks at specified times without manual intervention. Advantages include: unlike traditional load-triggered scaling, this application supports precise time-based scheduling, suitable for scenarios involving scaling up during peak periods and scaling down during off-peak periods. Compared to potentially erroneous scaling up during brief load increases, this invention, based on fixed-time triggering, ensures that scaling up meets business requirements. This application does not require external task scheduling tools; it leverages the native scheduling mechanism of the distributed management platform to achieve timed scaling up, improving compatibility and maintainability.

[0114] 2.1 This application expands the resources of compute nodes and task management nodes by combining the configuration resource information of the cluster deployment object. It modifies the configuration resource information of the cluster deployment object through the interface of the distributed management platform, updates the hardware resource configuration of the compute nodes and task management nodes based on the updated configuration resource information, and submits it to the preset interface of the resource management platform to trigger the restart of the runtime environment. 2.2 After resource adjustment, the platform operator redeploys the operator parallelism to ensure that the resource expansion is synchronized with the runtime environment. The advantages are: compared to manual expansion, this application is fully automated, requiring no manual intervention, thus improving operational efficiency. Compared to load-based expansion, this application supports more flexible resource management, allowing for fine-grained expansion processing of CPU, memory, and card slots.

[0115] 3.1 This application integrates platform interfaces (e.g., Flink REST API) to expand job parallelism without restarting the job. It adjusts the parallelism of operators through the platform interface, achieving seamless scaling of streaming computing tasks. 3.2 This application employs an incremental update strategy, adjusting only the operators that need expansion without affecting the running status of other operators. Advantages include: traditional scaling methods require restarting the runtime platform, while this invention's parallelism scaling does not require restarting the runtime platform or interrupting the job, maintaining uninterrupted data flow and ensuring business continuity. This application allows for fine-grained control of parallelism, avoiding resource waste caused by increasing the number of runtime platforms. This application supports independent scaling of different operators, improving job execution efficiency and reducing resource overhead.

[0116] 4.1 Expansion and contraction status management and rollback mechanism: This application can record the state before and after expansion and contraction. Before each expansion or contraction, the original configuration before expansion is saved, and the expansion history is stored in preset fields of the resource configuration file. 4.2 Automatic rollback: After the expansion time window ends, the original CPU, memory, and parallelism configurations can be automatically restored to avoid long-term occupation of computing resources. Advantages: Compared to manual rollback, this application supports automatic rollback, ensuring reasonable resource usage and avoiding prolonged occupation. Compared to automatic contraction, this application ensures that the job is restored to a predetermined state during contraction without affecting job stability. The historical records of this application can be used to analyze the effects of expansion and contraction, facilitating operation and maintenance optimization.

[0117] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0118] Based on the same inventive concept, this application also provides a timed capacity expansion device for implementing the timed capacity expansion method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more timed capacity expansion device embodiments provided below can be found in the limitations of the timed capacity expansion method described above, and will not be repeated here.

[0119] In one exemplary embodiment, such as Figure 5 As shown, a timed capacity expansion device is provided, including: a file acquisition module 502, a task generation module 504, and a capacity expansion processing module 506, wherein:

[0120] The file acquisition module 502 is used to acquire a custom resource configuration file; the resource configuration file includes a custom expansion time, expansion target, and expansion strategy;

[0121] The task generation module 504 is used to generate scheduled expansion tasks for the preset distributed management platform based on the resource configuration file; the scheduled expansion tasks are native expansion task resources of the distributed management platform.

[0122] The expansion processing module 506 is used to determine the expansion strategy and expansion target based on the scheduled expansion task when the distributed management platform determines that the expansion time has arrived, and to perform expansion processing on the expansion target according to the expansion strategy.

[0123] In one embodiment, the timed expansion device further includes a file generation module, which is used to determine a preset distributed management platform, obtain the custom resource definition of the distributed management platform; determine the expansion time, expansion target and expansion strategy; and generate a custom resource configuration file according to the format specification, expansion time, expansion target and expansion strategy defined by the custom resource definition.

[0124] In one embodiment, the task generation module 504 is further configured to parse configuration key information from the resource configuration file; generate a task configuration file for the scheduled expansion task based on the configuration key information; and submit the task configuration file to the distributed management platform so that the distributed management platform can generate the scheduled expansion task based on the task configuration file.

[0125] In one embodiment, the configuration key information includes expansion time, expansion target, resource configuration file name, and expansion strategy name; the task generation module 504 is also used to fill the expansion time extracted from the resource configuration file into the expansion time filling position in the task configuration file template; and to fill the expansion target, configuration file name, and expansion strategy name into the container startup command in the task configuration file template to obtain the task configuration file.

[0126] In one embodiment, the expansion processing module 506 is further configured to, when the expansion time recorded in the timed expansion task is reached, determine the expansion target, resource configuration file name and expansion strategy name recorded in the expansion task; read the corresponding resource configuration file through the resource configuration file name, and read the expansion strategy corresponding to the expansion strategy name from the read resource configuration file.

[0127] In one embodiment, the expansion processing module 506 is further configured to determine the running vehicle corresponding to the expansion target indicated by the timed expansion task; expand the hardware resources of the running vehicle according to the hardware resource expansion strategy in the expansion strategy; and expand the parallelism of the running vehicle according to the parallel expansion strategy in the expansion strategy.

[0128] In one embodiment, the expansion processing module 506 is further configured to delete the running carrier; and create a new running carrier according to the hardware resource expansion strategy in the expansion strategy, so that the running carrier has the hardware resources corresponding to the hardware resource expansion strategy.

[0129] In one embodiment, the expansion processing module 506 is further configured to modify the configuration information of the cluster deployment object of the distributed management platform according to the hardware resource expansion strategy in the expansion strategy; when the native controller of the distributed management platform detects that the configuration information of the cluster deployment object has changed, it creates a new running carrier according to the configuration information of the cluster deployment object.

[0130] In one embodiment, the expansion processing module 506 is further configured to adjust the operator parallelism of the new running vehicle while keeping the new running vehicle running, according to the parallel expansion strategy in the expansion strategy.

[0131] In one embodiment, the timed expansion device further includes an information recording module, which stores the pre-expansion specifications corresponding to the expansion target before expansion; and stores the post-expansion specifications corresponding to the expansion target after expansion is completed, and stores the expansion execution status.

[0132] In one embodiment, the timed expansion device further includes a rollback module, which is used to roll back the expanded specifications to the original specifications for the expansion target when the expansion time window ends.

[0133] Each module in the aforementioned timed capacity expansion device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0134] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data for timed expansion. The I / O interfaces allow the processor to exchange information with external devices. The communication interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a timed expansion method.

[0135] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0136] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0137] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0138] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0141] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A timed capacity expansion method, characterized in that, The method includes: Obtain a custom resource configuration file; the resource configuration file includes a custom expansion time, expansion target, and expansion strategy; Based on the resource configuration file, a scheduled expansion task for the preset distributed management platform is generated; the scheduled expansion task is the native expansion task resource of the distributed management platform. When the distributed management platform determines that the expansion time has arrived, it determines the expansion strategy and expansion target based on the scheduled expansion task, and performs expansion processing on the expansion target according to the expansion strategy.

2. The method according to claim 1, characterized in that, The method further includes: Determine the preset distributed management platform and obtain the custom resource definition of the distributed management platform; Determine the expansion timeline, expansion targets, and expansion strategies; A custom resource configuration file is generated based on the format specifications defined in the custom resource definition, the expansion time, the expansion target, and the expansion strategy.

3. The method according to claim 1, characterized in that, The step of generating a scheduled expansion task for a preset distributed management platform based on the resource configuration file includes: Extract key configuration information from the resource configuration file; Based on the aforementioned key configuration information, generate a task configuration file for the scheduled expansion task; The task configuration file is submitted to the distributed management platform so that the distributed management platform can generate a scheduled expansion task based on the task configuration file.

4. The method according to claim 3, characterized in that, The key configuration information includes expansion time, expansion target, resource configuration file name, and expansion strategy name; The step of generating a task configuration file for the scheduled expansion task based on the configured key information includes: The expansion time extracted from the resource configuration file will be filled into the expansion time field in the task configuration file template. Fill the container startup command field in the task configuration file template with the expansion target, the configuration file name, and the expansion strategy name to obtain the task configuration file.

5. The method according to claim 1, characterized in that, When the distributed management platform determines that the expansion time has arrived, it determines the expansion strategy and expansion target based on the scheduled expansion task, including: When the expansion time recorded in the scheduled expansion task is reached, the expansion target, resource configuration file name, and expansion strategy name recorded in the expansion task are determined. The resource configuration file corresponding to the resource configuration file name is read, and the expansion strategy corresponding to the expansion strategy name is read from the read resource configuration file.

6. The method according to claim 1, characterized in that, The step of scaling up the target node in the distributed cluster corresponding to the scaling target according to the scaling strategy includes: Determine the operating platform corresponding to the expansion target indicated by the timed expansion task; According to the hardware resource expansion strategy in the expansion strategy, the hardware resources of the running carrier are expanded; According to the parallel expansion strategy in the expansion strategy, the parallelism of the running carrier is expanded.

7. The method according to claim 6, characterized in that, The step of expanding the hardware resources of the running platform according to the hardware resource expansion strategy in the expansion strategy includes: Delete the aforementioned operating platform; According to the hardware resource expansion strategy in the expansion strategy, a new running platform is created so that the running platform has the hardware resources corresponding to the hardware resource expansion strategy.

8. The method according to claim 7, characterized in that, The step of creating a new runtime environment based on the hardware resource expansion strategy in the expansion strategy includes: Based on the hardware resource expansion strategy in the expansion strategy, the configuration information of the cluster deployment object of the distributed management platform is modified; When the native controller of the distributed management platform detects a change in the configuration information of the cluster deployment object, it creates a new runtime environment based on the configuration information of the cluster deployment object.

9. The method according to claim 8, characterized in that, The step of expanding the parallelism of the running platform according to the parallel expansion strategy in the expansion strategy includes: According to the parallel expansion strategy in the expansion strategy, the operator parallelism of the new running platform is adjusted while keeping the new running platform running.

10. The method according to claim 1, characterized in that, The method further includes: Before scaling up, store the original specifications corresponding to the scaling target. Once the expansion is complete, store the expanded specifications corresponding to the expansion target, and store the expansion execution status.

11. The method according to claim 10, characterized in that, The method further includes: When the expansion window ends, the expanded specifications will be rolled back to the original specifications for the expansion target.

12. A timed capacity expansion device, characterized in that, The device includes: The file acquisition module is used to acquire a custom resource configuration file; the resource configuration file includes a custom expansion time, expansion target, and expansion strategy; The task generation module is used to generate a scheduled expansion task for a preset distributed management platform based on the resource configuration file; the scheduled expansion task is a native expansion task resource of the distributed management platform. The expansion processing module is used to determine the expansion strategy and expansion target based on the timed expansion task when the distributed management platform determines that the expansion time has arrived, and to perform expansion processing on the expansion target according to the expansion strategy.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images