Method and system for realizing Flink on kubernetes application mode mirror image reuse based on minio

CN119025215APending Publication Date: 2024-11-26ZHEJIANG SHUXIN NETWORK CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310597932.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

[0009]现实开发人员进行任务开发时,经常需要对代码进行调试,也就是任务程序jar经常会发生变更,如果每次提交任务都去重新构建镜像,由于构建镜像是比较费时的操作,所以会浪费大量的时间

Benefits of technology

[0045] This invention optimizes the process of submitting tasks in the flink on kubernetes application mode and simplifies the submission steps. By reusing the template image, there is no need to build an image every time a task is submitted. Instead, the task jar is advanced before the jobmanager process and the taskmanager process are started. Download to the specified directory within the container. Not only can it save time and cost, but it can also reduce the technical requirements for development personnel and shield docker and kubernetes-related technical implementations, making it easier for developers to get started.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025215A_ABST
    Figure CN119025215A_ABST
Patent Text Reader

Abstract

The invention discloses a method for realizing Flink on kubernetes application mode mirror image reuse based on minio, and belongs to the field of big data component containerization deployment. The method comprises the following steps: creating a template mirror image; an flink task is constructed; according to the flink task, a pod arrangement file and an flink run submission command are generated; a task program jar and a dependency jar in the flink task are uploaded to a minio cluster; a command is submitted according to the flink run, and a pod is deployed in a kubernetes cluster; the method comprises the following steps of: arranging a file according to a pod, and creating an initContainer initialization container in the pod through a busybox mirror image; according to the pod arrangement file, creating a mainContainer main container in the pod through the template mirror image; the invention further discloses a system for achieving mirror image reuse in the Flink on kubernetes application mode on the basis of the minio, and the system is used for achieving mirror image reuse in the Flink on kubernetes application mode on the basis of the minio. According to the method, time cost can be saved, meanwhile, the technical requirements for development task personnel can be reduced, docker and kubernetes related technical implementation are shielded, and development personnel are easier to master.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of containerized deployment of big data components, and specifically includes a method and system for implementing Flink onkubernetes application mode image reuse based on minio. Background Art

[0002] As a representative technology of big data real-time computing, Flink is leading the way with its high performance, high throughput, low latency, perfect fault tolerance mechanism, and rich ecology. At the same time, with the continuous development of virtualization technology, Kubernetes, as an open source system for automating the deployment, expansion, and management of containerized applications, has gradually gained favor among more and more enterprises and developers.

[0003] The Flink v1.15 release supports native Kubernetes mode, which is further divided into Kubernetes session mode and Kubernetes application mode.

[0004] When submitting a task in kubernetes application mode, the path of the task program Jar package is limited to local mode, that is, only the Jar existing in the image can be specified, and it cannot be specified as a Jar in a shared storage such as hdfs. Therefore, each time a task is submitted, a new image needs to be built to include the jar in it.

[0005] The basic process of submitting tasks in kubernetes application deployment mode is as follows:

[0006] 1. The user manually builds the developed task program jar and dependent program jar into a basic image and uploads the image to the image warehouse;

[0007] 2. The user executes the flink run command to submit the task, specifying the image used for the task, the kubernetes user, the kubernetes permission file, the kubernetes pod orchestration yaml file, and other flink running parameters;

[0008] 3. Flink calls the kubernetes API to start the flinkjobmanager process and taskmanager process according to the parameters executed in the user startup command, and executes the user program.

[0009] In reality, when developers are developing tasks, they often need to debug the code, that is, the task program jar often changes. If you rebuild the image every time you submit a task, it will waste a lot of time because building the image is a time-consuming operation. Summary of the invention

[0010] The purpose of the present invention is to provide a method and system for realizing Flink on kubernetes application mode image reuse based on minio, which can not only save time cost but also reduce the technical requirements for development task personnel.

[0011] In order to solve the above technical problems, the present invention provides a method for implementing Flink on kubernetes application mode image reuse based on minio, comprising the following steps:

[0012] Create a template image;

[0013] Build a flink task; the flink task includes the task program jar, dependent jars and flink related running parameters;

[0014] Generate pod orchestration files and flink run submission commands based on the flink-related running parameters in the flink task;

[0015] Upload the task program jar and dependent jar in the flink task to the minio cluster;

[0016] According to the flink run submission command, deploy the pod in the kubernetes cluster;

[0017] According to the pod orchestration file, create an initContainer initialization container in the pod through the busybox image;

[0018] According to the pod orchestration file, create the mainContainer in the pod through the template image;

[0019] According to the flink run submission command, a temporary shared directory is generated for mounting the initContainer initialization container and the mainContainer main container;

[0020] Download the task program jar and dependent jar from the minio cluster to a temporary shared directory.

[0021] Preferably, the flink run submission command includes the address of the used kubernetes cluster and task program jar and dependent Jar in the container.

[0022] Preferably, according to the flink run submission command, the pod is deployed in the kubernetes cluster, specifically including the following steps:

[0023] Deploy the pod in the corresponding Kubernetes cluster according to the Kubernetes cluster used in the flink run submission command.

[0024] Preferably, according to the flink run submission command, a temporary shared directory for mounting the initContainer initialization container and the mainContainer main container is generated, which specifically includes the following steps:

[0025] According to the task program jar in the flink run submission command and the address of the dependent Jar in the container, a temporary shared directory for mounting the initContainer initialization container and the mainContainer main container is generated in the pod.

[0026] Preferably, downloading the task program jar and dependent jar from the minio cluster to a temporary shared directory specifically includes the following steps:

[0027] Use the wget tool to download the task program Jar and dependent Jar from the minio cluster to a temporary shared directory.

[0028] Preferably, the pod orchestration file includes a description of an initContainer initialization container and a description of a mainContainer main container.

[0029] Preferably, according to the pod orchestration file, an initContainer initialization container is created in the pod through the busybox image, specifically including the following steps:

[0030] According to the description of the initContainer initialization container in the pod orchestration file, create the initContainer initialization container of the corresponding description in the pod through the busybox image.

[0031] Preferably, according to the pod orchestration file, the mainContainer main container is created in the pod through the template image, specifically including the following steps:

[0032] According to the description of the mainContainer in the pod orchestration file, create the mainContainer with the corresponding description in the pod through the template image.

[0033] Preferably, the flink-related operating parameters include specifying a kubernetes user, image, and memory.

[0034] The present invention also provides a system for implementing Flink on kubernetes application mode image reuse based on minio, comprising:

[0035] Template image creation module, used to create template images;

[0036] The flink task building module is used to build a flink task; the flink task includes the task program jar, dependent jar and flink related running parameters;

[0037] The task processing module is used to generate pod orchestration files and flink run submission commands according to the flink-related running parameters in the flink task;

[0038] The upload module is used to upload the task program jar and dependent jar in the flink task to the minio cluster;

[0039] The pod deployment module is used to deploy pods in the kubernetes cluster according to the flink run submission command;

[0040] The initContainer initialization container creation module is used to create an initContainer initialization container in the pod through the busybox image according to the pod orchestration file;

[0041] The mainContainer main container creation module is used to create the mainContainer main container in the pod through the template image according to the pod orchestration file;

[0042] The temporary shared directory generation module is used to generate temporary shared directories for mounting the initContainer initialization container and the mainContainer main container according to the flink run submission command;

[0043] The download module is used to download the task program jar and dependent jar from the minio cluster to a temporary shared directory.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention optimizes the task submission process of the flink on kubernetes application mode, simplifies the submission steps, and reuses the template image, so that the image does not need to be built every time a task is submitted. Instead, the task jar is downloaded to a specified directory in the container in advance before the jobmanager process and the taskmanager process are started. This can not only save time costs, but also reduce the technical requirements for task development personnel, shield the technical implementation related to docker and kubernetes, and make it easier for developers to get started. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0047] Figure 1 This is an overall architecture diagram of a system for implementing Flink on kubernetes application mode image reuse based on minio in the present invention;

[0048] Figure 2 This is a schematic diagram of the process of constructing the initContainer initialization container and the mainContainer main container in Example 1;

[0049] Figure 3 The present invention is a flowchart of a method for implementing Flink on kubernetes application mode image reuse based on minio. DETAILED DESCRIPTION

[0050] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.

[0051] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0052] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0053] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0054] like Figure 3 As shown, the present invention provides a method for implementing Flink on kubernetes application mode image reuse based on minio, comprising the following steps:

[0055] Create a template image;

[0056] Build a flink task; the flink task includes the task program jar, dependent jars and flink related running parameters;

[0057] Generate pod orchestration files and flink run submission commands based on the flink-related running parameters in the flink task;

[0058] Upload the task program jar and dependent jar in the flink task to the minio cluster;

[0059] According to the flink run submission command, deploy the pod in the kubernetes cluster;

[0060] According to the pod orchestration file, create an initContainer initialization container in the pod through the busybox image;

[0061] According to the pod orchestration file, create the mainContainer in the pod through the template image;

[0062] According to the flink run submission command, a temporary shared directory is generated for mounting the initContainer initialization container and the mainContainer main container;

[0063] Download the task program jar and dependent jar from the minio cluster to a temporary shared directory.

[0064] Preferably, the flink run submission command includes the address of the used kubernetes cluster and task program jar and dependent Jar in the container.

[0065] Preferably, according to the flink run submission command, the pod is deployed in the kubernetes cluster, specifically including the following steps:

[0066] Deploy the pod in the corresponding Kubernetes cluster according to the Kubernetes cluster used in the flink run submission command.

[0067] Preferably, according to the flink run submission command, a temporary shared directory for mounting the initContainer initialization container and the mainContainer main container is generated, which specifically includes the following steps:

[0068] According to the task program jar in the flink run submission command and the address of the dependent Jar in the container, a temporary shared directory for mounting the initContainer initialization container and the mainContainer main container is generated in the pod.

[0069] Preferably, downloading the task program jar and dependent jar from the minio cluster to a temporary shared directory specifically includes the following steps:

[0070] Use the wget tool to download the task program Jar and dependent Jar from the minio cluster to a temporary shared directory.

[0071] Preferably, the pod orchestration file includes a description of an initContainer initialization container and a description of a mainContainer main container.

[0072] Preferably, according to the pod orchestration file, an initContainer initialization container is created in the pod through the busybox image, specifically including the following steps:

[0073] According to the description of the initContainer initialization container in the pod orchestration file, create the initContainer initialization container of the corresponding description in the pod through the busybox image.

[0074] Preferably, according to the pod orchestration file, the mainContainer main container is created in the pod through the template image, specifically including the following steps:

[0075] According to the description of the mainContainer in the pod orchestration file, create the mainContainer with the corresponding description in the pod through the template image.

[0076] Preferably, the flink-related operating parameters include specifying a kubernetes user, image, and memory.

[0077] The present invention also provides a system for implementing Flink on kubernetes application mode image reuse based on minio, comprising:

[0078] Template image creation module, used to create template images;

[0079] The flink task building module is used to build a flink task; the flink task includes the task program jar, dependent jar and flink related running parameters;

[0080] The task processing module is used to generate pod orchestration files and flink run submission commands according to the flink-related running parameters in the flink task;

[0081] The upload module is used to upload the task program jar and dependent jar in the flink task to the minio cluster;

[0082] The pod deployment module is used to deploy pods in the kubernetes cluster according to the flink run submission command;

[0083] The initContainer initialization container creation module is used to create an initContainer initialization container in the pod through the busybox image according to the pod orchestration file;

[0084] The mainContainer main container creation module is used to create the mainContainer main container in the pod through the template image according to the pod orchestration file;

[0085] The temporary shared directory generation module is used to generate temporary shared directories for mounting the initContainer initialization container and the mainContainer main container according to the flink run submission command;

[0086] The download module is used to download the task program jar and dependent jar from the minio cluster to a temporary shared directory.

[0087] In order to better illustrate the technical effect of the present invention, the present invention provides the following specific embodiments to illustrate the above technical process:

[0088] Example 1: A system for reusing Flink on kubernetes application mode images based on minio, and the overall architecture is described in combination with the overall architecture diagram, such as Figure 1 As shown, including:

[0089] taskSubmiter component:

[0090] The taskSubmiter component mainly includes the management of template images, the submission of flink tasks, and the management of related program jars and dependent jars. The webUI is mainly convenient for users to manage template images and submit tasks. After the flink task is submitted, the podbuilder component will dynamically generate the kubernetes pod orchestration file according to the parameters specified by the user submitting the flink task. The flinkClient client is used to generate the command to submit the task and actually submit the task to the kubernetes cluster for execution. The fileManager file manager is used to manage the jar, including upload and download related operations.

[0091] Kubernetes cluster:

[0092] After the flink task is submitted to the kubernetes cluster by the taskSubmiter component, kubernetes will start the jobmanager process and the pod corresponding to the taskmanager process according to the command to parse and execute the jar of the task. The pod contains initContainer (initialization container, which mainly downloads the task jar and synchronizes it to the main container) and mainContainer (main container, the container that actually executes the task logic).

[0093] Image warehouse image warehouse:

[0094] It is mainly used to store template images. When the template image is created, it will be saved here. At the same time, when Kubernetes starts the container, it will also download the image from the warehouse to the node of the Kubernetes cluster and then start the container.

[0095] A method for reusing Flink on kubernetes application mode images based on minio. The specific implementation steps are as follows: Figure 2 As shown, the following steps are included:

[0096] 1. The user creates a template image through the taskSubmiter component and uploads it to the image repository (if the template image already exists, this step can be skipped).

[0097] 2. The user submits a flink task. This process requires uploading the task program jar and dependent jar. At the same time, the flink-related running parameters (such as kubernetes user, image, memory, etc.) can be specified. The taskSubmiter component will dynamically generate the jobmanager process and the taskmanager process's pod orchestration file based on the kubernetes cluster according to the parameters specified by the user.

[0098] The orchestration file mainly includes the description of the initContainer initialization container and the mainContainer main container, including the images used by the two containers, the startup commands, etc. The startup command of the initContainer initialization container uses wget to download the corresponding jar that has been uploaded to the minio cluster. When generating the command, the server will be called to configure the minio API to generate the http download address of the corresponding jar, and then assembled into a download command.

[0099] 3. After the pod orchestration file is generated, the flinkclient module will dynamically generate the flink run submission command for execution based on the parameters submitted by the user. The command will specify the kubernetes cluster used, the address of the task program jar in the container, etc.;

[0100] 4. After receiving the request, the kubernetes cluster will start the pod corresponding to the jobmanager process. According to the pod orchestration file we generated, two containers will be specified in the file, namely the initContainer initialization container and the mainContainer main container;

[0101] initContainer is created using busybox image, and the startup command of initContainer is generated according to the parameters submitted by the task. The command uses wget tool to download the task program Jar and the required dependency Jar to the specified directory, and the directory is mounted as a temporary shared directory. The mainContainer also mounts the same temporary shared directory, so that the task program jar and the dependency jar are synchronized to the mainContainer, and then the jar address specified in the flink task submission instruction generated in the third step is truly effective. The initContainer task is destroyed after execution, and then the mainContainer is started. The image used by the mainContainer is the template image we created. After startup, the task program jar will be found for parsing and initialization operations, and then the corresponding number of taskmanager process pods will be pulled up in the kubernetes cluster according to the configured parallelism, etc. These pods will also have initContainer to download the corresponding jar, and then start the mainContainer to execute the task, and the process ends here.

[0102] The key technologies of the present invention are:

[0103] 1. Submit and run flink tasks based on kubernetes;

[0104] 2. Build a container template image based on Docker;

[0105] 3.Store and download task program jar and dependent jar based on minio;

[0106] 4. Download the program-related jar based on the busybox image and kubernetes initContainer (initialization container);

[0107] 5. Submit flink tasks and manage template images based on the taskSubmiter component;

[0108] As mentioned above, key technology 1: Kubernetes supports container deployment and rollback, which makes it easy to deploy applications in containers and provides ecosystem support such as scheduling and storage. In Kubernetes, pod is the smallest deployable computing unit that can be created and managed. Pod is a group of (one or more) containers. The jobmanager process and taskmanager process of Flink correspond to one pod in Kubernetes respectively. If a task contains one jobmanager process and four taskmanager processes, it will correspond to five pods in Kubernetes.

[0109] As mentioned above, key technology 2: Docker is an application container engine that allows users to package applications and related dependencies into an image. An image can be understood as an executable software package that contains everything needed to run an application.

[0110] As mentioned above, key technology 3: minio is a high-performance object storage component that supports distributed deployment and can ensure data reliability. It can generate an http address for a specific file and download the corresponding file by accessing the address through a browser or tools such as wget. The task jar and dependent jar developed by the user can be uploaded to minio for storage, or downloaded by accessing the relevant address.

[0111] As mentioned above, key technology 4: kubernetes initContainer is a special container. Each pod can contain multiple initContainers. They are executed before the main container in the pod and stop after execution. Its characteristics can be used to perform some main container initialization actions. For example, you can use initContainer to download jar and then synchronize the downloaded jar to the application container through temporary shared storage. The busybox image is a lightweight image that contains many common Linux tools, and the tools in it can be used to initialize the container.

[0112] As above key technology 5:

[0113] The taskSubmiter component is an application service developed based on Java. It provides a web interface to facilitate users to submit Flink tasks and manage template images.

[0114] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules, modules or units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units, modules or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0115] The units may or may not be physically separated, and the components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0116] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0117] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present invention are executed. It should be noted that the above-mentioned computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above.

[0118] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for implementing Flink on kubernetes application mode image reuse based on minio, characterized in that: The following steps are involved: Create a template image; Build a flink task; the flink task includes the task program jar, dependent jars and flink related running parameters; Generate pod orchestration files and flink run submission commands based on the flink-related running parameters in the flink task; Upload the task program jar and dependent jar in the flink task to the minio cluster; According to the flink run submission command, deploy the pod in the kubernetes cluster; According to the pod orchestration file, create an initContainer initialization container in the pod through the busybox image; According to the pod orchestration file, create the mainContainer in the pod through the template image; According to the flink run submission command, a temporary shared directory is generated for mounting the initContainer initialization container and the mainContainer main container; Download the task program jar and dependent jar from the minio cluster to a temporary shared directory.

2. According to the method for implementing Flink on kubernetes application mode image reuse based on minio according to claim 1, it is characterized in that: The flink run submission command includes the address of the used kubernetes cluster and task program jar and dependent jar in the container.

3. The method for implementing Flink on kubernetes application mode image reuse based on minio according to claim 2, characterized in that: According to the flink run submission command, deploy the pod in the kubernetes cluster, which includes the following steps: Deploy the pod in the corresponding Kubernetes cluster according to the Kubernetes cluster used in the flink run submission command.

4. The method for implementing Flink on kubernetes application mode image reuse based on minio according to claim 2, characterized in that: According to the flink run submission command, a temporary shared directory for mounting the initContainer initialization container and the mainContainer main container is generated. The specific steps include: According to the task program jar in the flink run submission command and the address of the dependent Jar in the container, a temporary shared directory for mounting the initContainer initialization container and the mainContainer main container is generated in the pod.

5. The method for implementing Flink on kubernetes application mode image reuse based on minio according to claim 1, characterized in that: Download the task program jar and dependent jar from the minio cluster to a temporary shared directory. The specific steps include: Use the wget tool to download the task program Jar and dependent Jar from the minio cluster to a temporary shared directory.

6. The method for implementing Flink on kubernetes application mode image reuse based on minio according to claim 1, characterized in that: The pod orchestration file includes a description of the initContainer initialization container and a description of the mainContainer main container.

7. The method for implementing Flink on kubernetes application mode image reuse based on minio according to claim 6, characterized in that: According to the pod orchestration file, create an initContainer initialization container in the pod through the busybox image, which includes the following steps: According to the description of the initContainer initialization container in the pod orchestration file, create the initContainer initialization container of the corresponding description in the pod through the busybox image.

8. The method for implementing Flink on kubernetes application mode image reuse based on minio according to claim 6, characterized in that: According to the pod orchestration file, create the mainContainer in the pod through the template image, which includes the following steps: According to the description of the mainContainer in the pod orchestration file, create the mainContainer with the corresponding description in the pod through the template image.

9. The method for implementing Flink on kubernetes application mode image reuse based on minio according to claim 1, characterized in that: The flink-related operating parameters include specifying the kubernetes user, image, and memory.

10. A system for implementing Flink on kubernetes application mode image reuse based on minio, used to implement the method for implementing Flink on kubernetes application mode image reuse based on minio as described in any one of claims 1-9, characterized in that: include: Template image creation module, used to create template images; The flink task building module is used to build a flink task; the flink task includes the task program jar, dependent jar and flink related running parameters; The task processing module is used to generate pod orchestration files and flink run submission commands according to the flink-related running parameters in the flink task; The upload module is used to upload the task program jar and dependent jar in the flink task to the minio cluster; The pod deployment module is used to deploy pods in the kubernetes cluster according to the flink run submission command; The initContainer initialization container creation module is used to create an initContainer initialization container in the pod through the busybox image according to the pod orchestration file; The mainContainer main container creation module is used to create the mainContainer main container in the pod through the template image according to the pod orchestration file; The temporary shared directory generation module is used to generate temporary shared directories for mounting the initContainer initialization container and the mainContainer main container according to the flink run submission command; The download module is used to download the task program jar and dependent jar from the minio cluster to a temporary shared directory.