Method and device for generating arrangement file, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a method for generating an arrangement file, belongs to the technical field of big data edge computing, and can improve the efficiency of deploying jobs to clusters. Comprising the following steps: adding a resource list parameter in a configuration item of a job source code of a target application program, the resource list parameter comprises one or more of the following items: a path parameter of mapping a job to be deployed into a target cluster, a mount naming parameter of a persistent volume in a target application program, and a persistent volume naming parameter of the persistent volume in the target cluster; defining a persistent volume declaration parameter of a mapping relationship between the mounting naming parameter and the persistent volume naming parameter; modifying a target cluster deployment method in the job source code according to the resource list parameter, and generating a new job and a configuration file including the resource list parameter; the target job is introduced through the configuration file in the target application program, an arrangement file is generated, the arrangement file is used for deploying the target job into the target cluster, and the target job is a new job or a continuous transmission job after the breakpoint of the new job.

Description

Technical Field

[0001] The present application relates to the field of big data edge computing technology, and in particular to a method, device, electronic device and storage medium for generating an orchestration file. Background Art

[0002] Currently, the flink job, a processing framework that runs a big data processing engine based on a container cluster management system (Kubernetes, K8S), is a common cluster deployment method.

[0003] In the related art, the storage of the checkpoint file of the Flink job requires the specification of configuration parameters. Different configuration parameters correspond to different configuration files. The configuration file is also a resource. Different resources need to be packaged into image files of different versions. Therefore, when the job is deployed to the cluster, it strongly relies on the configuration file in the application to generate the orchestration file, resulting in low efficiency in deploying the job to the cluster. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for generating an orchestration file, so as to solve the problem of low efficiency in deploying jobs to a cluster.

[0005] To solve the above technical problems, the embodiments of the present application are implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for generating an orchestration file, comprising: adding a resource list parameter in a configuration item of a job source code of a target application, the resource list parameter comprising one or more of the following: a path parameter for mapping the job to be deployed to a target cluster, a mount naming parameter for a persistent volume in the target application, a persistent volume naming parameter for a persistent volume in the target cluster, and a persistent volume declaration parameter defining a mapping relationship between the mount naming parameter and the persistent volume naming parameter; the target application is deployed in a cluster corresponding to the target application; according to the resource list parameter, the target cluster deployment method in the job source code is modified to generate a new job and configuration file including the resource list parameter; the target job is introduced through the configuration file in the target application, and an orchestration file is generated, the orchestration file being used to deploy the target job to the target cluster, the target job being the new job or a resumed job after a breakpoint of the new job.

[0007] In a second aspect, an embodiment of the present application provides a device for generating an orchestration file, comprising: a configuration module, used to add resource list parameters in the configuration items of the job source code of the target application, the resource list parameters including one or more of the following: path parameters of the job to be deployed mapped to the target cluster, mount naming parameters of the persistent volume in the target application, persistent volume naming parameters of the persistent volume in the target cluster, and persistent volume declaration parameters defining the mapping relationship between the mount naming parameters and the persistent volume naming parameters; the target application is deployed in the cluster corresponding to the target application;

[0008] A generation module, used to modify the target cluster deployment method in the job source code according to the resource list parameters, and generate a new job and configuration file including the resource list parameters;

[0009] An orchestration module is used to introduce a target job through the configuration file in the target application and generate an orchestration file, wherein the orchestration file is used to deploy the target job to the target cluster, and the target job is the new job or a resumed job after the breakpoint of the new job.

[0010] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being used to call and execute the computer program from the memory to implement the above-mentioned method for generating an arrangement file.

[0011] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program can be executed by a processor to implement the above-mentioned method for generating an arrangement file.

[0012] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the above-mentioned method for generating an arrangement file.

[0013] In a sixth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the above-mentioned method for generating an arrangement file is implemented.

[0014] By adopting the technical solution of the embodiment of the present application, resource list parameters are added to the configuration items of the job source code of the target application, and the resource list parameters include one or more of the following: path parameters of the job to be deployed mapped to the target cluster, mount naming parameters of the persistent volume in the target application, persistent volume naming parameters of the persistent volume in the target cluster, and persistent volume declaration parameters that define the mapping relationship between the mount naming parameters and the persistent volume naming parameters; the target application is deployed in the cluster corresponding to the target application; according to the resource list parameters, the target cluster deployment method in the job source code is modified to generate a new job and configuration file including the resource list parameters; the target job is introduced through the configuration file in the target application, and an orchestration file is generated, and the orchestration file is used to deploy the target job to the target cluster, and the target job is a new job or a resumed job after a new job breakpoint. Among them, the target application introduces a new job that adds resource configuration parameters in the job source code of the job to be deployed. The configuration file generated by the new job also includes resource inventory parameters. According to the path parameters, mount naming parameters, persistent volume naming parameters and persistent volume declaration parameters in the resource configuration parameters, the mapping relationship between the new job in the target application and the target cluster is determined. The target application can introduce the target job through the configuration file, where the target job can be a new job or a resumed job. The orchestration file can be dynamically generated. The orchestration file is used to deploy the target job to the target cluster. The target job can be deployed without packaging the configuration file and the target job multiple times, which can improve the efficiency of job deployment to the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate one or more embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 is a schematic flow chart of a method for generating an arrangement file according to an embodiment of the present application;

[0017] Figure 2 is a structural diagram of a method for generating an arrangement file according to an embodiment of the present application;

[0018] Figure 3 is a schematic flow chart of a method for generating an arrangement file according to another embodiment of the present application;

[0019] Figure 4 is a schematic block diagram of a device for generating an arrangement file according to an embodiment of the present application;

[0020] Figure 5 It is a schematic diagram of the hardware structure of a device for generating a layout file according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] The embodiments of the present application provide a method, device, electronic device and storage medium for generating an orchestration file, so as to solve the problem of low efficiency in deploying jobs to a cluster.

[0022] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0023] The method for generating an arrangement file provided in the embodiment of the present application can be executed by an electronic device, or by software installed in the electronic device. Specifically, the electronic device can be a terminal device or a server device. Among them, the terminal device can include a smart phone, a laptop computer, a smart wearable device, a vehicle terminal, etc., and the server device can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0024] In the following, in conjunction with the accompanying drawings, a method for generating an arrangement file provided in an embodiment of the present application is described in detail through specific embodiments and application scenarios.

[0025] Figure 1 A schematic flow chart of a method for generating an arrangement file provided by an embodiment of the present invention is shown, and the method comprises the following steps:

[0026] S102, adding resource list parameters to the parameter configuration items of the job source code of the target application.

[0027] The resource list parameters include one or more of the following: path parameters that map the job to be deployed to the target cluster, mount naming parameters of the persistent volume in the target application, persistent volume naming parameters of the persistent volume in the target cluster, and persistent volume declaration parameters that define the mapping relationship between the mount naming parameters and the persistent volume naming parameters; the target application is deployed in the cluster corresponding to the target application.

[0028] The target application can be pre-deployed in the cluster corresponding to the target application, that is, the target application is deployed in a resource management component (Plain Old Data, POD) of the cluster. The target application is a real-time computing platform that can reference the Flink job source code of Apache and rewrite the job source code in the parameter configuration items of the job source code.

[0029] The job source code is specifically a Flink job written using Apache, that is, the Apache Flink source code. For example, for Apache Flink source code download, the version number can be 1.16. By configuring the project management tool software Maven version 3.6.1 to download the relevant dependency packages, the job source code is successfully compiled; the compiled job source code is made into a flink-kubernetes-1.16.jar package; the target application modifies the job source code in the jar package by referencing the flink-kubernetes-1.16.jar package, using the gateway as the entry point, and determines the modified jar package as a new job.

[0030] A persistent volume (PV) is an abstraction of storage resources, defining storage as a resource that can be used by container applications in a cluster. A persistent volume claim (PVC) is an application for storage resources, that is, an application for persistent volume resources, which can apply for a specific storage capacity size and access mode.

[0031] The job to be deployed can be a Flink job, i.e., a flinkSql job; the target cluster can be a k8s cluster. The added resource list parameters include: the path parameter of the job to be deployed mapped to the target cluster, that is, the path in the container of the target cluster specified by the Flink job on the target application side; the mount naming parameter of the persistent volume in the target application, that is, the name of the persistent volume specified on the target application side; the persistent volume naming parameter of the persistent volume in the target cluster, that is, the name of the persistent volume specified in the target cluster, the mount naming parameter should be consistent with the persistent volume naming parameter; the persistent volume declaration parameter that defines the mapping relationship between the mount naming parameter and the persistent volume naming parameter, that is, the declaration information of the persistent volume resource in the target cluster, which is used to declare that the mount naming parameter of the persistent volume name on the target application side and the persistent volume naming parameter of the persistent volume name in the target cluster can be mapped to, and the mapping relationship means that the mount naming parameter and the persistent volume naming parameter are consistent.

[0032] Optionally, the target application page displays a resource list, and the user fills in or modifies parameter information in the resource list through the target application page. At the same time, the resource list parameters in the parameter configuration items of the job source code of the target application are also updated synchronously.

[0033] S104, modifying the target cluster deployment method in the job source code according to the resource list parameters, and generating a new job and configuration file including the resource list parameters.

[0034] The configuration file PodTemplate file is an important part of the deployment resources. It is used to improve the deployed instance, namely POD. The file can supplement the content of the pod. It should be understood that the target cluster can be a POD. POD is a collection of containers, including one or more containers. PodTemplate can define the structure, configuration and initialization script of the deployed POD, rather than the entire pod structure. As an example, regarding the problem of deploying instances when resuming jobs from breakpoints, it is necessary to specify the storage path of the checkpoint file of the resumed job, namely the path parameter. The path parameter strongly depends on the predefined YAML format configuration file podTemplate.yaml file to enrich the orchestration file content according to the resource list parameters and the configuration file, so that POD can be quickly created and managed during the deployment process. It can be seen that through the resource list parameters in the configuration file, the target application can introduce the job to be deployed into the target cluster.

[0035] By adding resource list parameters to the job source code of the job to be deployed, that is, adding codes corresponding to the resource list parameters to the job source code to form a new job, a configuration file including the resource list parameters can be generated when the new job is running.

[0036] The job source code contains the task content of the job, including the parameters for deploying the job. According to the added resource list parameters, the method for deploying the target cluster in the job source code is updated to generate a new job including the new method for deploying the target cluster. Specifically, there are resource list parameters in the configuration file, and there are also newly added resource list parameters in the job source code of the new job. When the new job is introduced to the target cluster, the new job can be deployed by modifying other parameters in the new job, without modifying the data in the configuration file again, and without over-reliance on the configuration file.

[0037] S106, introducing the target job through the configuration file in the target application, and generating an orchestration file, where the orchestration file is used to deploy the target job to the target cluster.

[0038] The target job is a new job or a resumed job after a breakpoint in the new job.

[0039] The resource list parameters in the configuration file of the target application can be used to determine the location of the target job in the container of the target application, and then introduce the target job. The configuration file and the target job in the target application are used to generate an orchestration file, which is used to deploy the target job to the target cluster to form a Flink cluster, thereby improving the execution efficiency and flexibility of the target job deployment. The location of the target job in the container of the target application includes the location of the checkpoint file used for the breakpoint resumption of the target job in the container.

[0040] It should be understood that the orchestration file includes which node the cluster container is scheduled to, when to trigger and end the task, and how to handle exceptions in the task of the job. For example, when an exception occurs in a new job, the location of the resume job after the breakpoint can be located through the configuration file and the resource list parameters added in the new job source code. By modifying the resume job obtained after the new job, the resume job can be introduced into the target cluster for deployment without repackaging the configuration file and other operations. The resource list parameters in the configuration file can be mapped to the location of the target job and the location of the target job in the target cluster. Therefore, the target application can introduce a new job into the target cluster through the configuration file, and can also introduce the resume job after the breakpoint of the new job into the target cluster, that is, the orchestration file includes the configuration file and the target job. The target job can be a new job or a resume job after the breakpoint of the new job is modified.

[0041] By adopting the technical solution of the embodiment of the present application, resource list parameters are added to the configuration items of the job source code of the target application, and the resource list parameters include one or more of the following: path parameters of the job to be deployed mapped to the target cluster, mount naming parameters of the persistent volume in the target application, persistent volume naming parameters of the persistent volume in the target cluster, and persistent volume declaration parameters that define the mapping relationship between the mount naming parameters and the persistent volume naming parameters; the target application is deployed in the cluster corresponding to the target application; according to the resource list parameters, the target cluster deployment method in the job source code is modified to generate a new job and configuration file including the resource list parameters; the target job is introduced through the configuration file in the target application, and an orchestration file is generated, and based on the orchestration file, the target job is deployed to the target cluster, and the target job is a new job or a resumed job after a new job breakpoint. Among them, the target application introduces a new job that adds resource configuration parameters in the job source code of the job to be deployed. The configuration file generated by the new job also includes resource inventory parameters. According to the path parameters, mount naming parameters, persistent volume naming parameters and persistent volume declaration parameters in the resource configuration parameters, the mapping relationship between the new job in the target application and the target cluster is determined. The target application can introduce the target job through the configuration file, where the target job can be a new job or a resumed job. The orchestration file can be dynamically generated. The orchestration file is used to deploy the target job to the target cluster. The target job can be deployed without packaging the configuration file and the target job multiple times, which can improve the efficiency of job deployment to the cluster.

[0042] In one embodiment, to add resource list parameters to the configuration items of the job source code of the target application (ie, S102), the following step A may be performed:

[0043] Step A: Add resource list parameters to the source code of the configuration item of the job source code through the gateway of the cluster corresponding to the target application.

[0044] Specifically, in the source code of the configuration item of the job source code, a new resource list parameter is created, and the added resource list parameter corresponds to the finally generated orchestration file.

[0045] Among them, add resource list parameters, including:

[0046] Determine the mount path in the target cluster that the job is mapped to based on the path parameter. The path parameter can be expressed as: KUBERNETES_POD_VOLUMEMOUNTS_MOUNTPATH.

[0047] Specifically, there is a path in the target cluster from which the Flink job on the target application side is mapped. For example, there is a mapping relationship between the path of the Flink job in the target application and the path in the target cluster to which the target application is submitted. The path parameter is a parameter for storing the mapping relationship. According to the path parameter, the mount path in the container of the target cluster where the Flink job POD is mapped is specified.

[0048] Determine the name of the persistent volume in the target application based on the mount naming parameter. The mount naming parameter can be expressed as: KUBERNETES_POD_VOLUMEMOUNTS_NAME.

[0049] Specifically, the name of the persistent volume used to access the Flink job on the target application side.

[0050] The name of the persistent volume in the target cluster according to the persistent volume naming parameter. The persistent volume naming parameter can be expressed as: KUBERNETES_POD_VOLUMS_NAME.

[0051] Specifically, the persistent volume is named after the persistent volume defined in the target cluster.

[0052] Determine the mapping between the naming of the Flink job in the target application and the naming in the target cluster based on the persistent volume claim parameters. The persistent volume claim parameters can be expressed as: KUBERNETES_POD_VOLUMS_PERSITENTVOLUMECLAIM_CLAIMNAME.

[0053] Persistent volume declaration parameters, defined in the cluster POD corresponding to the target application and outside the POD in the target cluster to be deployed, are used to define the persistent volume declaration information of the target cluster in the persistent volume. The persistent volume declaration parameters can declare the mapping relationship between the naming of the Flink job in the target application and the naming in the target cluster. According to the above-mentioned consistent mount naming parameters and persistent volume naming parameters, the two are mapped through the persistent volume declaration parameters so that the target application can introduce the Flink job into the target cluster and open up the relationship between the Flink job and the target cluster. Generate a configuration file based on the added resource list parameters.

[0054] It should be noted that each resource list parameter corresponds to the information in the orchestration file that is ultimately used to deploy the target cluster. As an example, the correspondence between the newly added resource list parameters and the parameters in the orchestration file is shown in the following table:

[0055]

[0056] In this embodiment, by adding a resource list parameter to the configuration item of the target application's job source code, the target application can be associated with the Flink job to be deployed, and can also be associated with the information in the target cluster where the Flink job is deployed, so as to facilitate the subsequent deployment of the Flink job to the target cluster.

[0057] In one embodiment, the target cluster deployment method in the job source code is modified to generate a new job and configuration file including resource list parameters (ie, S104), and the following steps B1-B2 may be performed:

[0058] Step B1: In the cluster descriptor source code of the job source code, define the request resource object of the persistent volume, and assign values ​​to the request resource object according to the persistent volume declaration parameters in the resource list parameters.

[0059] The cluster descriptor is a class in the job source code for the internal target cluster deployment method. The cluster descriptor is represented by KubernetesClusterDescriptor, and the internal target cluster deployment can be represented by deployClusterInternal. To rewrite the internal target cluster deployment method, rewrite the deployClusterInternal method in the KubernetesClusterDescriptor class.

[0060] First, in the cluster descriptor source code of the job source code, define a persistent volume request resource object, that is, the user's request object for persistent volume resources; obtain the name of the request resource object through the Flink configuration parameter flinkConfig.get method, and assign values ​​to the properties of the request resource object according to the obtained name. The input parameter of the get method is the name of the persistent volume declared in the persistent volume declaration parameter in the resource list parameter. The get method is a request method used to obtain data in the network.

[0061] Step B2: In the cluster descriptor source code of the job source code, define a persistent volume object, and assign a name to the persistent volume object according to the persistent volume naming parameter in the resource inventory parameter; and assign a declaration to the persistent volume object according to the persistent volume declaration parameter in the resource inventory parameter.

[0062] In the cluster descriptor source code of the job source code, first define a persistent volume object, then assign a value to the naming attribute of the persistent volume object, and then assign a value to the declaration attribute of the persistent volume object according to the persistent volume declaration parameter in step B1. The method for assigning a value to the naming attribute of the persistent volume object can also be obtained through the flinkConfig.get method, and the input parameter of this method is the persistent volume naming parameter in the resource list parameter.

[0063] Step B3: In the cluster descriptor source code of the job source code, define a persistent volume mount object, and assign a name to the persistent volume mount object according to the mount naming parameter in the resource list parameter; and assign a path to the persistent volume mount object according to the path parameter in the resource list parameter.

[0064] In the cluster descriptor source code of the job source code, define the persistent volume mount object, and then assign a value to the name of the persistent volume mount object and a value to the mount path of the persistent volume mount object. Both of these are obtained through the flinkConfig.get method, and the input parameters of this method are the mount name parameter and path parameter in the resource list parameter.

[0065] Step B4: assign the persistent volume named object and the persistent volume mount named object of the persistent volume to the configuration file, and generate a new job and configuration file including resource list parameters.

[0066] By defining a persistent volume object, the deployment location of the job in the target cluster can be specified; by defining a persistent volume mount object, the storage location information of the job on the target application side can be determined. By defining a persistent volume object and a persistent volume mount object, the location information of the job on the target application side and the location information mapped to the target cluster can be determined, thereby forming a new target cluster deployment method and generating a new job. And the persistent volume naming object and the persistent volume mount naming object of the persistent volume are assigned to the configuration file. When running a new job to generate a configuration file, based on the assigned persistent volume naming object and persistent volume mount naming object and the corresponding information, the location of the new job can be determined through the configuration file. Specifically, the persistent volume object and the persistent volume mount object in step B2 and step B3 can be assigned to the configuration file respectively. For example, it is implemented by the podTemplate.getPodWithoutMainContainer().getSpec().getVolumes() method add(volume) and by the podTemplate.getMainContainer().getVolumeMounts().add(volumeMount) method.

[0067] In this embodiment, by defining persistent volume objects and persistent volume mount objects in the job source code, and by using the flinkConfig.get method to obtain parameter information in the resource list parameters to assign values ​​to the defined objects, a new target cluster deployment method is formed, and a new job including resource list parameters is generated. The new job defines the resource list parameters and is then referenced by the target application. The resource list parameters are assigned to the configuration file through the running of the new job, and a configuration file containing the resource list parameters is generated. According to the newly added resource list parameters, the new job can be introduced into the corresponding target cluster by the target application through the configuration file, thereby improving the efficiency of deploying new jobs.

[0068] In one embodiment, the target job is introduced through the configuration file in the target application and the orchestration file is generated (ie, S106), and the following step B41 may be performed:

[0069] Step B41: When a job interruption occurs in the new job, the new job is modified through the target application to generate a target job.

[0070] A new job refers to a job that has resource list parameters configured in the job source code. When a new job is introduced into an application and the transmission of the new job is interrupted, the content of the new job is modified through the target application, and the new job that is subsequently transmitted after the modification is used as a resumed job. The target job includes a resumed job and a new job that does not require modification of the job source code.

[0071] Step B42: Introduce the target job through the configuration file in the target application, and determine the orchestration file according to the target job and the configuration file.

[0072] Since the configuration file is assigned with the resource list parameter information, it can correspond to the specified location of the new job before the target job is modified, and can also determine the location of the target job. According to the modified target job and configuration file of the new job, the target application can determine the location of the target job in the container of the target application through the newly added resource list parameters in the configuration file, and then introduce the target job into the target cluster for deployment. The orchestration files used to deploy the target cluster are the target job and configuration file. When the new job is not interrupted or modified, the new job can also be introduced into the target cluster as the target job.

[0073] It should be noted that the job manager JobManager is the core of task management and scheduling in the Flink cluster. The orchestration file can be dynamically generated through the resource list parameters. Specifically, when a new job is deployed to the target cluster, a Flink cluster is generated. When the new job is interrupted, the job manager class in the Flink cluster creates the detailed content of the job manager as the entry. The specific method is as follows: Use the KubernetesJobManagerFactory.buildKubernetesJobManagerSpecification method as the entry, deploy and load the POD resource list, and dynamically generate the orchestration file. Specifically, if the resource list parameters corresponding to the resume job in the target job change, since the resume job is running, the information in the resource list parameters of the new job corresponding to the resume job is assigned to the configuration file. Therefore, the information in the configuration file can determine the location of the resume job. If the name of the persistent volume corresponding to the resume job is changed, the parameters in the configuration file are automatically changed. The orchestration file including the resume job and the configuration file is dynamically generated. There is no need to rebuild different versions of the image due to changes in the configuration file, that is, to repackage the resume job and the configuration file, and the target application can also run normally.

[0074] As an example, when a new job is introduced into the target application, if a problem occurs in the new job, causing the new job to be interrupted, such as a problem with the library table in the new job, etc., since the persistent volume object and the persistent volume mount object are defined in the new job source code and assigned to the configuration file, when the new job problem is solved, the storage location designated for the target job can be determined according to the resource list parameters in the configuration file. Among them, the target job can be a resume job in the checkpoint file, and the checkpoint file is a file that saves the interrupted job after the new job is interrupted. The target application continues to introduce the modified job, i.e., the target job, into the target cluster without packaging the configuration file and the target job, which can improve the efficiency of deploying the job to the cluster and prevent the problem of breakpoint resumption.

[0075] In this embodiment, the persistent volume object and persistent volume mount object defined in the job source code are assigned to the configuration file. There is no need to pre-define the configuration file. According to the resource list parameters in the configuration file, the target cluster can find the location of the target job and continue to execute the target job. It can be seen that the target application can introduce new jobs into the corresponding target cluster through the configuration file. When a problem occurs in the new job, the location of the target job can be determined through the configuration file, and the modified target job can be continued to be introduced into the target cluster, thereby preventing the need to repackage the configuration file after the job breakpoint, and improving the efficiency of job deployment to the cluster.

[0076] In one embodiment, after the target job is deployed to the target cluster (ie, S106), the following step D1 may be performed:

[0077] Step D1: Monitor the jobs after they are deployed to the target cluster and verify the new jobs through the plug-in.

[0078] Verify whether the target job deployed in the target cluster can run normally. Specifically, monitor the running target job in the operation and maintenance center of the target application. Through the task name in the target cluster plug-in, that is, kubectledit deploy-n${namespace}flinkSql task name, you can check whether the content generated by the dynamic orchestration file meets expectations, and whether the target job file saved by breakpoint continuation is generated on the persistent volume, and verify whether the target job meets expectations. If it does not meet expectations, you can adjust the resource configuration parameters. If it meets expectations, it means that there is no problem with the content in the configuration file and the new job source code.

[0079] In this embodiment, the accuracy of deploying the target job to the target cluster is improved by verifying the running status of the target job.

[0080] In one embodiment, the resource list parameters are added to the job source code (ie, S104), and the following step E1 may be performed:

[0081] Step E1: According to the resource list parameters, add the definition code of the resource list parameters in the static initialization block of the configuration item source code of the job source code.

[0082] Specifically, according to the resource list parameters, add the resource list parameter definition code in the static initialization block of the target cluster configuration option KubernetesConfigOptions source code. First, for the path parameter, define the mount path mounthPath of the persistent volume volumeMounts in the target cluster kubernetes container through the keyword key in the configuration option ConfigOptions, which can specify the flink job POD mapped to the mount path in the target cluster container.

[0083] Secondly, for the mount naming parameter, the name of the persistent volume volumeMounts mounted in the kubernetes container is defined through the key of ConfigOptions. The name of the mounted persistent volume can be specified. The name should be consistent with the value of the persistent volume naming parameter in the target cluster so that the mount relationship can correspond.

[0084] Then, the persistent volume naming parameter defines the name of the persistent volume volumes in the kubernetes container through the key of ConfigOptions, which is used to define the name of a persistent volume in the target cluster;

[0085] Finally, the persistent volume claim parameter defines the persistent volume claim information of volumes in the kubernetes container through the key of ConfigOptions, persitentVolumeClaim.claimName, and defines the claim information of the persistent volume resource of the target cluster;

[0086] In this embodiment, by adding resource list parameter definition code in the static initialization block of the configuration item source code of the job source code, the defined resource list parameters can be used to modify the deployment method of the target cluster in the class of the cluster descriptor.

[0087] In one embodiment, the method further includes asynchronously executing the job, and specifically executing the following step F:

[0088] Step F: Obtain the persistent volume through the persistent volume declaration parameters, determine the target cluster, obtain the target application with the same persistent volume naming parameters as the target cluster, determine the target job according to the persistent volume declaration parameters, and deploy the target job to the target cluster through the target application, where one target application can correspond to multiple target jobs and target clusters.

[0089] Apply for persistent volumes through persistent volume declaration parameters, determine the target cluster and target application with consistent persistent volume declaration parameters, and specify the target job to be mapped to the target cluster. You can specify the target cluster to be deployed for the target job, introduce multiple target jobs into the corresponding multiple target clusters, and each job does not interfere with each other. In other words, a target application can execute multiple target jobs asynchronously, each target job corresponds to a configuration file and target cluster, and the target job and target cluster executed are different due to different resource list parameters.

[0090] In this embodiment, by asynchronously executing multiple target jobs in a target program, the efficiency of deploying the target jobs to the target cluster is improved.

[0091] In summary, if Figure 2The figure shows a structural diagram of a method for generating an orchestration file, in which the job source code of the job to be deployed is compiled and referenced through the target application, and the configuration items in the job source code are optimized by adding resource list parameters, and the newly added resource list parameters are assigned to the configuration file on the generated target application side, replacing the resource list parameters in the target application, and generating a target cluster deployment method that can dynamically orchestrate the target job. The target job and configuration file after the resource list parameters are added on the target application side can be used to asynchronously execute the target job. After the target job is deployed to the target cluster, the target job is monitored and processed to verify that the target job is running normally.

[0092] Figure 3 is a schematic flow chart of a method for generating an arrangement file according to another embodiment of the present application, such as Figure 3 As shown, the method comprises the following steps:

[0093] S301, creating a persistent volume independent of a target cluster and a target application through persistent volume declaration information, and determining resource inventory parameters based on the persistent volume.

[0094] S302, download the Apache Flink source code, determine the job source code by compiling, and introduce the job source code into the target application.

[0095] S303, modifying the job source code through the gateway of the target application.

[0096] S304, creating a new resource list parameter in the configuration item of the job source code, and adding the code of the resource list parameter.

[0097] S305, defining resource list parameters in the cluster descriptor source code of the job source code, and modifying the method of deploying the target cluster to form a new job.

[0098] S306, when the new job is running, a configuration file is generated, and resource list parameters are assigned to the configuration file.

[0099] S307, in the target application, introduce the new job into the target cluster through the configuration file.

[0100] S308, when a breakpoint occurs in the new job, the new job is modified to form a resume job, and the resume job is continuously introduced into the target cluster through the resource list parameters assigned in the configuration file.

[0101] It should be noted that the resource list parameters in the configuration file are consistent with the resource list parameters in the resume job.

[0102] The specific process from S301 to S308 has been described in detail in the above embodiment and will not be repeated here.

[0103] By adopting the technical solution of the embodiment of the present application, resource list parameters are added to the configuration items of the job source code of the target application, and the resource list parameters include one or more of the following: path parameters of the job to be deployed mapped to the target cluster, mount naming parameters of the persistent volume in the target application, persistent volume naming parameters of the persistent volume in the target cluster, and persistent volume declaration parameters that define the mapping relationship between the mount naming parameters and the persistent volume naming parameters; the target application is deployed in the cluster corresponding to the target application; according to the resource list parameters, the target cluster deployment method in the job source code is modified to generate a new job and configuration file including the resource list parameters; the target job is introduced through the configuration file in the target application, and an orchestration file is generated, and based on the orchestration file, the target job is deployed to the target cluster, and the target job is a new job or a resumed job after a new job breakpoint. Among them, the target application introduces a new job that adds resource configuration parameters in the job source code of the job to be deployed. The configuration file generated by the new job also includes resource inventory parameters. According to the path parameters, mount naming parameters, persistent volume naming parameters and persistent volume declaration parameters in the resource configuration parameters, the mapping relationship between the new job in the target application and the target cluster is determined. The target application can introduce the target job through the configuration file, where the target job can be a new job or a resumed job. The orchestration file can be dynamically generated. The orchestration file is used to deploy the target job to the target cluster. The target job can be deployed without packaging the configuration file and the target job multiple times, which can improve the efficiency of job deployment to the cluster.

[0104] In summary, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0105] The above is a method for generating an arrangement file provided in an embodiment of the present application. Based on the same idea, an embodiment of the present application also provides a device for generating an arrangement file.

[0106] Figure 4 Schematic diagram of a device for generating a layout file according to an embodiment of the present invention. Figure 4 As shown, the device for generating the arrangement file includes: a configuration module 41, a generation module 42, and an arrangement module 43:

[0107] Configuration module 41, used to add resource list parameters in the configuration items of the job source code of the target application, the resource list parameters including one or more of the following: path parameters of the job to be deployed mapped to the target cluster, mount naming parameters of the persistent volume in the target application, persistent volume naming parameters of the persistent volume in the target cluster, and persistent volume declaration parameters defining the mapping relationship between the mount naming parameters and the persistent volume naming parameters; the target application is deployed in the cluster corresponding to the target application;

[0108] A generating module 42, used to modify the target cluster deployment method in the job source code according to the resource list parameters, and generate a new job and configuration file including the resource list parameters;

[0109] The orchestration module 43 is used to introduce the target job through the configuration file in the target application and generate an orchestration file, which is used to deploy the target job to the target cluster. The target job includes a new job or a resumed job after a new job breakpoint.

[0110] In one embodiment, the configuration module 41 is specifically used to add resource list parameters in the source code of the configuration item of the job source code through the gateway of the cluster corresponding to the target application;

[0111] The parameters for adding resource lists include:

[0112] Determine the mount path in the target cluster where the job is mapped based on the path parameters;

[0113] Determine the name of the mounted persistent volume in the target application based on the mount naming parameters;

[0114] Determine the name of the persistent volume in the target cluster based on the persistent volume naming parameters;

[0115] Determine the mapping between the job's naming in the target application and its naming in the target cluster based on the persistent volume claim parameters.

[0116] In one embodiment, the generating module 42 is specifically used to define a request resource object of a persistent volume in the cluster descriptor symbol source code of the job source code, and assign a value to the request resource object according to the persistent volume declaration parameter in the resource list parameter;

[0117] In the cluster descriptor source code of the job source code, define a persistent volume object, and assign a value to the persistent volume object name according to the persistent volume naming parameter in the resource inventory parameter; and assign a value to the persistent volume object declaration according to the persistent volume declaration parameter in the resource inventory parameter;

[0118] In the cluster descriptor source code of the job source code, define the persistent volume mount object, and assign a value to the name of the persistent volume mount object according to the mount naming parameter in the resource list parameter; and assign a value to the path of the persistent volume mount object according to the path parameter in the resource list parameter;

[0119] Assign the persistent volume named object and the persistent volume mount named object of the persistent volume to the configuration file, and generate a new job and configuration file including resource list parameters.

[0120] In one embodiment, the orchestration module 43 is specifically used to modify the new job through the target application program to generate a target job when a job interruption occurs in the new job;

[0121] The target job is introduced through the configuration file in the target application, and the orchestration file is determined according to the target job and the configuration file.

[0122] In one embodiment, the device is also used to monitor the target job after it is deployed in the target cluster, and verify the target job through the plug-in.

[0123] In one embodiment, the configuration module 41 is further configured to add definition codes of resource list parameters in a static initialization block of a configuration item source code of the job source code according to the resource list parameters.

[0124] By adopting the technical solution of the embodiment of the present application, resource list parameters are added to the configuration items of the job source code of the target application, and the resource list parameters include one or more of the following: path parameters of the job to be deployed mapped to the target cluster, mount naming parameters of the persistent volume in the target application, persistent volume naming parameters of the persistent volume in the target cluster, and persistent volume declaration parameters that define the mapping relationship between the mount naming parameters and the persistent volume naming parameters; the target application is deployed in the cluster corresponding to the target application; according to the resource list parameters, the target cluster deployment method in the job source code is modified to generate a new job and configuration file including the resource list parameters; the target job is introduced through the configuration file in the target application, and an orchestration file is generated, and based on the orchestration file, the target job is deployed to the target cluster, and the target job is a new job or a resumed job after a new job breakpoint. Among them, the target application introduces a new job that adds resource configuration parameters in the job source code of the job to be deployed. The configuration file generated by the new job also includes resource inventory parameters. According to the path parameters, mount naming parameters, persistent volume naming parameters and persistent volume declaration parameters in the resource configuration parameters, the mapping relationship between the new job in the target application and the target cluster is determined. The target application can introduce the target job through the configuration file, where the target job can be a new job or a resumed job. The orchestration file can be dynamically generated. The orchestration file is used to deploy the target job to the target cluster. The target job can be deployed without packaging the configuration file and the target job multiple times, which can improve the efficiency of job deployment to the cluster.

[0125] Those skilled in the art should understand that Figure 4 The arrangement file generation device in can be used to implement the arrangement file generation method described above, and the detailed description thereof should be similar to the description of the method part above, and will not be repeated here to avoid tediousness.

[0126] Based on the same technical concept, an embodiment of the present application further provides an electronic device, which is used to execute the above-mentioned method for generating an arrangement file. Figure 5 A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call a computer program stored in the memory 530 and executable on the processor 510 to perform the following steps:

[0127] Add resource list parameters to the configuration items of the target application's job source code. The resource list parameters include one or more of the following: path parameters that map the job to be deployed to the target cluster, mount naming parameters of the persistent volume in the target application, persistent volume naming parameters of the persistent volume in the target cluster, and persistent volume declaration parameters that define the mapping relationship between the mount naming parameters and the persistent volume naming parameters; the target application is deployed in the cluster corresponding to the target application;

[0128] According to the resource list parameters, modify the target cluster deployment method in the job source code to generate a new job and configuration file including the resource list parameters;

[0129] The target job is introduced through the configuration file in the target application, and an orchestration file is generated. The orchestration file is used to deploy the target job to the target cluster. The target job includes a new job or a resumed job after a new job breakpoint.

[0130] By adopting the technical solution of the embodiment of the present application, resource list parameters are added to the configuration items of the job source code of the target application, and the resource list parameters include one or more of the following: path parameters of the job to be deployed mapped to the target cluster, mount naming parameters of the persistent volume in the target application, persistent volume naming parameters of the persistent volume in the target cluster, and persistent volume declaration parameters that define the mapping relationship between the mount naming parameters and the persistent volume naming parameters; the target application is deployed in the cluster corresponding to the target application; according to the resource list parameters, the target cluster deployment method in the job source code is modified to generate a new job and configuration file including the resource list parameters; the target job is introduced through the configuration file in the target application, and an orchestration file is generated, and based on the orchestration file, the target job is deployed to the target cluster, and the target job is a new job or a resumed job after a new job breakpoint. Among them, the target application introduces a new job that adds resource configuration parameters in the job source code of the job to be deployed. The configuration file generated by the new job also includes resource inventory parameters. According to the path parameters, mount naming parameters, persistent volume naming parameters and persistent volume declaration parameters in the resource configuration parameters, the mapping relationship between the new job in the target application and the target cluster is determined. The target application can introduce the target job through the configuration file, where the target job can be a new job or a resumed job. The orchestration file can be dynamically generated. The orchestration file is used to deploy the target job to the target cluster. The target job can be deployed without packaging the configuration file and the target job multiple times, which can improve the efficiency of job deployment to the cluster.

[0131] The specific execution steps can refer to the various steps of the above-mentioned embodiment of the method for generating the arrangement file, and can achieve the same technical effect. To avoid repetition, they will not be described here.

[0132] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals or other devices except terminals.

[0133] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, switch buttons, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0134] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0135] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0136] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned embodiment of the method for generating an arrangement file is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0137] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0138] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the method for generating an arrangement file, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0139] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0140] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the processor is used to run the program or instructions to implement the various processes of the above-mentioned product recommendation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0141] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0143] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for generating a layout file, characterized in that: The method comprises: Add resource list parameters to the configuration items of the job source code of the target application, wherein the resource list parameters include one or more of the following: path parameters of the job to be deployed mapped to the target cluster, mount naming parameters of the persistent volume in the target application, persistent volume naming parameters of the persistent volume in the target cluster, and persistent volume declaration parameters defining a mapping relationship between the mount naming parameters and the persistent volume naming parameters; the target application is deployed in the cluster corresponding to the target application; According to the resource list parameters, modify the target cluster deployment method in the job source code to generate a new job and configuration file including the resource list parameters; A target job is introduced through the configuration file in the target application, and an orchestration file is generated. The orchestration file is used to deploy the target job to the target cluster. The target job is the new job or a resumed job after the breakpoint of the new job.

2. The method according to claim 1, characterized in that The adding of resource list parameters in the configuration item of the job source code of the target application program includes: Adding the resource list parameter to the source code of the configuration item of the job source code through the gateway of the cluster corresponding to the target application; Wherein, adding the resource list parameters includes: Determine, according to the path parameter, a mount path for mapping the job to the target cluster; Determining the name of the mounted persistent volume in the target application according to the mount naming parameter; Determine the name of the persistent volume in the target cluster according to the persistent volume naming parameter; A mapping relationship between a name of the job in the target application and a name of the job in the target cluster is determined according to the persistent volume declaration parameters.

3. The method according to claim 1, characterized in that The method of modifying the target cluster deployment method in the job source code to generate a new job and configuration file including the resource list parameters includes: In the cluster descriptor source code of the job source code, define a request resource object of the persistent volume, and assign a value to the request resource object according to the persistent volume declaration parameter in the resource list parameter; In the cluster descriptor source code of the job source code, a persistent volume object is defined, and a name of the persistent volume object is assigned according to the persistent volume naming parameter in the resource inventory parameter; and a declaration of the persistent volume object is assigned according to the persistent volume declaration parameter in the resource inventory parameter; In the cluster descriptor source code of the job source code, a persistent volume mount object is defined, and a name of the persistent volume mount object is assigned according to the mount naming parameter in the resource list parameter; and a path of the persistent volume mount object is assigned according to the path parameter in the resource list parameter; The persistent volume object and the persistent volume mount object of the persistent volume are assigned to the configuration file to generate the new job including the resource list parameters and the configuration file.

4. The method according to claim 3, characterized in that The step of introducing the target job through the configuration file in the target application and generating an orchestration file comprises: When the new job is interrupted, modifying the new job through the target application to generate the target job; The target job is introduced through the configuration file in the target application, and an orchestration file is determined according to the target job and the configuration file.

5. The method according to claim 1, characterized in that After the target job is deployed to the target cluster, the method further includes: The target job is monitored after being deployed in the target cluster, and the target job is verified through a plug-in.

6. The method according to claim 1, characterized in that The adding of resource list parameters to the configuration items of the job source code of the target application program also includes: According to the resource list parameters, definition codes of the resource list parameters are added in the static initialization block of the configuration item source code of the job source code.

7. A device for generating a layout file, characterized in that: The device comprises: A configuration module, configured to add resource list parameters to the configuration items of the job source code of the target application, wherein the resource list parameters include one or more of the following: path parameters for mapping the job to be deployed to the target cluster, mount naming parameters for the persistent volume in the target application, persistent volume naming parameters for the persistent volume in the target cluster, and persistent volume declaration parameters defining a mapping relationship between the mount naming parameters and the persistent volume naming parameters; the target application is deployed in the cluster corresponding to the target application; A generation module, used to modify the target cluster deployment method in the job source code according to the resource list parameters, and generate a new job and configuration file including the resource list parameters; An orchestration module is used to introduce a target job through the configuration file in the target application and generate an orchestration file, wherein the orchestration file is used to deploy the target job to the target cluster, and the target job is the new job or a resumed job after the breakpoint of the new job.

8. An electronic device, characterized in that: The method comprises a processor and a memory electrically connected to the processor, wherein the memory stores a computer program, and the processor is used to call and execute the computer program from the memory to implement a method for generating an arrangement file as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium is used to store a computer program, and the computer program can be executed by a processor to implement a method for generating a layout file as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements a method for generating a layout file as claimed in any one of claims 1 to 6.

Images