Method and device for dynamically generating Spark port access in Jupyter container

By dynamically generating nodePort ports and binding Services when Jupyter container Pod generation, the problem of not being able to use the yarn-client mode in the Jupyter container is solved, and interactive development of Spark cluster and the utilization of distributed computing capabilities is realized, which is simple and efficient in operation.

CN113094150BActive Publication Date: 2024-11-08SHANGHAI ZHONGTONGJI NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110360061.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-04-02
Publication Date
2024-11-08
Estimated Expiration
2041-04-02

AI Technical Summary

Technical Problem

The Jupyter container cannot meet the needs of running Spark programs directly using yarn-client mode, which will not be able to utilize the distributed computing power of the Spark cluster and the inability to meet the needs of the client's interactive development output.

Method used

在Jupyter容器Pod生成时动态生成多个不重复的nodePort端口,并基于这些端口生成绑定Pod的Service,保持port、targetPort、nodePort一致,从而在Jupyter容器内直接使用Spark on yarn的yarn-client模式运行Spark作业,获取交互式输出。

Benefits of technology

It implements the yarn-client mode of running Spark on yarn in a Jupyter container, which meets the needs of interactive development, is simple to operate and takes effect in real time, and users only need to add one line of configuration.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113094150B_ABST
    Figure CN113094150B_ABST
Patent Text Reader

Abstract

The present invention relates to a method and device for dynamically generating Spark port access in a Jupyter container, belonging to the field of communication technology. When a Jupyter container is generated, a non-repetitive Service NodePort is automatically and dynamically generated, and the driver.port and blockManager.port of a spark program are bound to it, so that a spark cluster can access the driver end of the Jupyter container in a k8s cluster. For the original spark program, only one line of configuration needs to be added, and the operation is simple and effective in real time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and in particular relates to a method and device for dynamically generating Spark port access in a Jupyter container. Background Art

[0002] After Jupyter is containerized using k8s, due to the internal and external isolation characteristics of the k8s cluster, the existing technology cannot meet the requirements of running spark programs in yarn-client mode inside the jupyter container.

[0003] For example, in the prior art, the Jupyter container cannot directly use the yarn-client mode, and can only run the spark program in local or cluster mode. The local mode cannot use the distributed computing capabilities of the spark cluster, and the cluster mode can only be viewed entirely on the spark cluster, which cannot meet the client's interactive development output requirements. Summary of the invention

[0004] The present invention provides a method and device for dynamically generating Spark port access in a Jupyter container, so as to solve the technical problem that the Jupyter container cannot satisfy the direct use of the yarn-client mode.

[0005] The technical solution provided by the present invention is as follows:

[0006] In one aspect, a method for dynamically generating Spark port access in a Jupyter container includes:

[0007] When starting to generate the Jupyter container Pod, multiple non-duplicate nodePort ports are dynamically generated;

[0008] Based on the nodePort port, generate a service bound to the jupyter pod, and keep the port, targetPort, and nodePort consistent;

[0009] Based on keeping consistent port, targetPort, and nodePort, you can directly use the yarn-client mode of spark on yarn in the jupyter container to run the spark job and get interactive output.

[0010] Optionally, the dynamically generated multiple non-repetitive nodePort ports include:

[0011] Use k8s-client to generate k8s cluster pod through yaml file;

[0012] According to the port global occupancy record table, query the available NodePort that has not been used, and poll and select a suitable nodeport from the available nodeports that have not been used.

[0013] Optionally, based on the nodePort port, generating a service bound to the jupyter pod and keeping the port, targetPort, and nodePort consistent include:

[0014] Based on the nodePort port, generate the Service bound to the jupyter container pod through yaml, and keep the port, targetPort, and nodePort completely consistent.

[0015] Optionally, based on maintaining consistent port, targetPort, and nodePort, the spark job is run directly in the jupyter container using the yarn-client mode of spark on yarn and interactive output is obtained, including:

[0016] The nodePort port on each node of the k8s cluster is proxied to the corresponding service service of the jupyter container pod, so that the external client of the k8s cluster can directly access the service entrance;

[0017] Run Spark jobs using the yarn-client mode of Spark on YARN directly in the Jupyter container and get interactive output.

[0018] Optionally, the nodePort port on each node of the k8s cluster is proxied to the corresponding service service of the jupyter container pod, so that the external client of the k8s cluster can directly access the service entrance, including:

[0019] Spark's Driver runs on the k8s Pod, maps the targetPort to the Port through the Service, and further maps it to the nodePort, opening up the entrance for external clients to access the k8s cluster, so that clients outside the k8s cluster can directly access the service entrance.

[0020] Optionally, the spark job is run directly in the jupyter container using the yarn-client mode of spark on yarn and interactive output is obtained, including:

[0021] Use Jupyter Pod as the client to initialize SparkContext and communicate with the Spark cluster;

[0022] Apply for executor resources from yarn's ResourceManager through Spark's ApplicationMaster. After AM starts the Executor, the allocated Executor managed by yarn NodeManager is reversely registered with the corresponding driver end.

[0023] The local Driver communicates with the applied executor container to schedule each Task; and the Executor reports the running status of each Task to the Driver, realizing interactive development in yarn-client mode.

[0024] In yet another aspect, a device for dynamically generating Spark port access in a Jupyter container includes: a processor, and a memory connected to the processor;

[0025] The memory is used to store a computer program, and the computer program is used to execute at least any of the above methods for dynamically generating Spark port access in a Jupyter container;

[0026] The processor is used to call and execute the computer program in the memory.

[0027] The beneficial effects of the present invention are:

[0028] The method and device for dynamically generating Spark port access in a Jupyter container provided by an embodiment of the present invention dynamically generates multiple non-repeating nodePort ports when starting to generate a jupyter container Pod; based on the nodePort port, a service bound to the jupyter pod is generated, and the port, targetPort, and nodePort are kept consistent; based on the consistent port, targetPort, and nodePort, the spark job is directly run in the jupyter container using the yarn-client mode of spark on yarn, and interactive output is obtained. The present invention automatically and dynamically generates non-repeating Service NodePort when generating a jupyter container, and binds the driver.port and blockManager.port of the spark program to it, so that the spark cluster can access the driver end of the jupyter container in the k8s cluster. For the original spark program, only one line of configuration needs to be added, which is simple to operate and takes effect in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 A flowchart of a method for dynamically generating Spark port access in a Jupyter container provided by an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of the principle structure of a method for dynamically generating Spark port access in a Jupyter container provided by an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of a device structure for dynamically generating Spark port access in a Jupyter container provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] To make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0034] Jupyter is an open source software, open standards and services for developing interactive computing across multiple programming languages. It can be applied to the whole process of computing, development, document writing, running code and displaying results. It is often used as an online development IDE for Python and machine learning developers. After the Jupyter component is deployed in a container, it can be used more lightweight and completely resource isolated.

[0035] k8s is an open source system for automatically deploying, scaling and managing containerized applications. Pod is the smallest deployable computing unit created and managed in k8s. Each jupyter container depends on pod generation. Service is an abstract method for exposing applications on a group of Pods as network services. Each service can be bound to the pod with the corresponding label through the selector label to provide services such as TCP for the pod.

[0036] Spark's yarn-client mode: One of the spark on yarn running modes. In this mode, the SparkDriver will run on the client, and only apply to the yarn's ResourceManager for executor resources through Spark's ApplicationMaster. After that, the local Driver client is responsible for communicating with all the requested executor containers to schedule tasks. The client cannot be closed. After the final result aggregation task is completed, closing the terminal is equivalent to killing the entire spark application. This mode is suitable for interactive local spark job development.

[0037] In the prior art, the Jupyter container cannot directly use the yarn-client mode, and can only run the spark program in local or cluster mode. The local mode cannot use the distributed computing capabilities of the spark cluster, and the cluster mode can only be viewed on the spark cluster, which cannot meet the needs of client interactive development output. In other words, the existing spark on yarn yarn-client mode cannot meet the way of directly running spark jobs in an independent k8s cluster. Since the k8s cluster is isolated from the outside, it cannot meet the needs of interactive development of spark jobs on an interactive IDE such as jupyter.

[0038] Based on this, an embodiment of the present invention provides a method for dynamically generating Spark port access in a Jupyter container.

[0039] Figure 1 A flowchart of a method for dynamically generating Spark port access in a Jupyter container provided by an embodiment of the present invention; Figure 2 A schematic diagram of the principle structure of a method for dynamically generating Spark port access in a Jupyter container provided in an embodiment of the present invention.

[0040] See also Figure 1 , Figure 2 The method provided by the embodiment of the present invention may include the following steps:

[0041] S11. When starting to generate the Jupyter container Pod, multiple non-duplicate nodePort ports are dynamically generated.

[0042] In some embodiments, optionally, multiple non-repetitive nodePort ports are dynamically generated, including:

[0043] Use k8s-client to generate k8s cluster pod through yaml file;

[0044] According to the port global occupancy record table, query the available NodePort that has not been used, and poll and select the applicable nodeport from the available nodeports that have not been used.

[0045] For example, when starting to generate a jupyter container Pod, multiple non-repetitive k8s cluster nodeports are automatically and dynamically generated.

[0046] S12. Based on the nodePort port, generate a service bound to the jupyter pod and keep the port, targetPort, and nodePort consistent; that is, the consistency between the cluster internal port and the external access port.

[0047] In some embodiments, optionally, it includes: based on the nodePort port, generating a Service bound to the jupyter container pod through yaml, and keeping the port, targetPort, and nodePort completely consistent.

[0048] For example, generate a Service bound to the jupyter container pod based on the nodeport, and keep the port, targetPort, and nodePort completely consistent, that is, the consistency between the cluster internal port and the external access port.

[0049] S13. Based on keeping consistent port, targetPort, and nodePort, use the yarn-client mode of spark on yarn directly in the jupyter container to run the spark job and obtain interactive output.

[0050] In some embodiments, optionally, based on maintaining consistent port, targetPort, and nodePort, the spark job is run directly in the jupyter container using the yarn-client mode of spark on yarn, and interactive output is obtained, including:

[0051] The nodePort port on each node of the k8s cluster is proxied to the corresponding service service of the jupyter container pod, so that the external client of the k8s cluster can directly access the service entrance;

[0052] Run Spark jobs using the yarn-client mode of Spark on YARN directly in the Jupyter container and get interactive output.

[0053] In some embodiments, optionally, it includes: Spark's Driver runs on the k8s Pod, maps the targetPort to the Port through the Service, and further maps it to the nodePort, so as to open up the entrance for external clients to access the inside of the k8s cluster, so that the external clients of the k8s cluster can directly access the service entrance.

[0054] In some embodiments, the optional steps include: using Jupyter Pod as the client to initialize SparkContext and communicate with the Spark cluster; applying for executor resources from yarn's ResourceManager through Spark's ApplicationMaster; after AM starts the Executor, the allocated Executor managed by yarn NodeManager is reversely registered to the corresponding Driver; the local Driver communicates with the applied executorcontainer to schedule each Task; and the Executor reports the running status of each Task to the Driver, thereby realizing interactive development in yarn-client mode.

[0055] In the embodiment of the present invention, the system automatically generates multiple unique NodePort ports in the k8s cluster that can be bound, which is transparent to the user. The user only needs to add a line of sparkConf configuration to the spark program started in the original yarn-client mode to add the binding port attribute, and it can be run in the jupyter container to take effect in real time, and the interactive spark job data output can be seen in the log, which is simple and efficient.

[0056] For example, to start a spark program in yarn-client mode in a Jupyter container, you only need to configure sparkConf to add the following properties after setMaster("yarn") (theoretically, it can support enough nodePorts):

[0057] set("spark.driver.bindAddress","0.0.0.0")

[0058] set("spark.driver.host","k8s cluster master node ip")

[0059] set("spark.driver.port","nodePort1")

[0060] set("spark.blockManager.port","nodePort2").

[0061] Optionally, when initializing SparkContext in the jupyter container, add the configuration sparkConf port property and set it to the corresponding selected multiple NodePorts, that is, the spark program can be run normally in yarn-client mode and interactive output can be obtained.

[0062] The method for dynamically generating Spark port access in a Jupyter container provided by an embodiment of the present invention is transparent to users. Users only need to add a line of binding port attributes added by sparkConf configuration to the spark program started in the original yarn-client mode, and the program can be run in the jupyter container to take effect in real time. Interactive spark job data output can be seen in the log, which is simple and efficient.

[0063] Based on a general inventive concept, an embodiment of the present invention further provides a device for dynamically generating Spark port access in a Jupyter container.

[0064] Figure 3For a schematic diagram of a device structure for dynamically generating Spark port access in a Jupyter container provided by an embodiment of the present invention, please refer to Figure 3 , a device for dynamically generating Spark port access in a Jupyter container provided by an embodiment of the present invention includes: a processor 31, and a memory 32 connected to the processor.

[0065] The memory 32 is used to store a computer program, and the computer program is at least used for the method of dynamically generating Spark port access in a Jupyter container recorded in any one of the above embodiments;

[0066] The processor 31 is used to call and execute the computer program in the memory.

[0067] Based on a general inventive concept, an embodiment of the present invention further provides a storage medium.

[0068] A storage medium stores a computer program. When the computer program is executed by a processor, each step of the method for dynamically generating Spark port access in a Jupyter container is implemented.

[0069] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0070] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0071] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0072] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0073] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0074] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0075] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0076] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0077] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0078] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for dynamically generating Spark port access in a Jupyter container, characterized in that: include: When starting to generate the Jupyter container Pod, multiple non-duplicate nodePort ports are dynamically generated; Based on the nodePort port, generate a service bound to the jupyter pod, and keep the port, targetPort, and nodePort consistent, including: based on the nodePort port, generate a Service bound to the jupyter container pod through yaml, and keep the port, targetPort, and nodePort completely consistent; Based on keeping consistent port, targetPort, and nodePort, use the yarn-client mode of spark onyarn directly in the jupyter container to run the spark job and get interactive output; Based on keeping consistent port, targetPort, and nodePort, the spark job is run directly in the jupyter container using the yarn-client mode of sparkon yarn and interactive output is obtained, including: The nodePort port on each node of the k8s cluster is proxied to the corresponding service service of the jupyter container pod, so that the external client of the k8s cluster can directly access the service entrance, including: Spark's Driver runs on the k8s Pod, maps the targetPort to the Port through the Service, and further maps it to the nodePort, so that the external client can access the entrance inside the k8s cluster, so that the external client of the k8s cluster can directly access the service entrance; Use the yarn-client mode of spark on yarn directly in the jupyter container to run the spark job and get interactive output, including: using Jupyter Pod as the client to initialize SparkContext and communicate with the Spark cluster; applying for executor resources from yarn's ResourceManager through Spark's ApplicationMaster. After AM starts the Executor, the allocated Executor managed by yarn NodeManager is reversely registered to the corresponding Driver end; the local Driver communicates with the applied executor container to schedule each Task; and the Executor reports the running status of each Task to the Driver end, realizing interactive development in yarn-client mode.

2. The method according to claim 1, characterized in that The dynamically generated multiple non-repetitive nodePort ports include: Use k8s-client to generate k8s cluster pod through yaml file; According to the port global occupancy record table, query the available NodePort that has not been used, and poll and select a suitable nodeport from the available nodeports that have not been used.

3. A device for dynamically generating Spark port access in a Jupyter container, characterized in that: include: A processor, and a memory connected to the processor; The memory is used to store a computer program, and the computer program is used to at least execute the method for dynamically generating Spark port access in a Jupyter container as described in any one of claims 1 to 2 above; The processor is used to call and execute the computer program in the memory.