Distributed parallel computing method and system based on Juser

Abstract

The invention provides a distributed parallel computing method and system based on JUPter, and belongs to the field of high-performance computing, message passing technologies and parallel computing. The problem that an existing interactive computing environment is difficult to directly support large-scale distributed parallel computing is solved. According to the main technical scheme, the method comprises the following steps of: developing a Juser kernel supporting MPI parallel computing; creating a JUPter plug-in, providing an intuitive configuration interface, and allowing a user to customize MPI cluster settings; a cooperation mechanism of a root node and a non-root node is realized; according to the method, the MPI code can be directly written and operated in the Juputer Notebook environment, so that the MPI code can be directly written and operated. The main technical effects include improvement of computational efficiency, simplification of parallel programming learning and application processes, enhancement of system flexibility and expandability, and improvement of user experience. According to the innovative technology, advanced tools and protocols such as IPython, Juser, ZMQ and MPI are fused, and new possibilities are brought to the fields of scientific calculation, data analysis and the like.

Description

Technical Field

[0001] The technical field of the present invention includes the field of high-performance computing and the field of parallel programming, and specifically relates to a Jupyter-based distributed parallel computing method and system. Background Art

[0002] As data volumes and computational complexity increase, interactive environments running on a single machine are facing performance bottlenecks. Traditional single-threaded or single-process computing is no longer sufficient for processing large data sets and computationally intensive tasks. Parallel computing is crucial for increasing computational speed and processing power.

[0003] Seamlessly integrating parallel computing with interactive environments is a current challenge. While some solutions attempt to combine Jupyter with distributed computing frameworks, such as IPython parallel and Dask, which are currently used to implement parallel computing in Jupyter Notebooks, these approaches still have the following shortcomings: they require additional configuration and a dedicated development environment, increasing the difficulty of use; they lack flexibility, making it difficult to dynamically adjust computing resources based on specific tasks; and they provide a poor user experience, requiring users to understand the underlying details of parallel programming, which increases the difficulty of interaction.

[0004] Therefore, the present invention hopes to propose a new design idea to seamlessly connect MPI parallel computing in the Jupyter Notebook environment, achieving the goal of user-friendliness without requiring additional API learning, allowing users to write and run MPI code directly in the Jupyter Notebook page, and realizing distributed parallel computing without user perception. Summary of the Invention

[0005] This invention aims to implement a kernel supporting MPI parallel computing within the Jupyter environment. This innovation breaks the traditional boundary between parallel computing and interactive programming. This approach allows users to write and run MPI code directly within the familiar Jupyter Notebook interface, without the need for additional configuration or a specialized development environment. By seamlessly combining the powerful communication capabilities of MPI with the interactivity of Jupyter Notebook, users can perform both single-machine programming and distributed parallel computing in the same environment.

[0006] The present invention provides a Jupyter-based distributed parallel computing method, the specific steps are as follows:

[0007] S1 creates a Jupyter kernel that supports MPI functionality. The kernel is responsible for executing the code entered by the user in the Jupyter Notebook and returning the results.

[0008] S2 Select and configure the kernel in Jupyter Notebook;

[0009] The root node in S3 MPI is responsible for distributing tasks and data to other processes, receiving and broadcasting code, executing code, collecting results and returning them to the front end;

[0010] In S4 MPI, the non-root node that receives instructions and data from the root node and returns the calculation results executes the broadcast code and returns the results;

[0011] S5 repeats S3 and S4 until the front end terminates the kernel.

[0012] In step S1, a kernel is created, specifically including:

[0013] S101: Create a kernel that supports MPI functions based on the IPython project and implement an interactive Python environment that supports MPI functions;

[0014] S102: Modify the kernel configuration information, including changing the kernel's display name and startup method, and start the kernel using mpiexe so that it can be managed by Jupyter.

[0015] S103: Create a Jupyter plug-in to configure the kernel and add configuration options to the overall Jupyter configuration, including: developing the configuration options of the plug-in, including configuring the number of MPI processes and configuring host information; monitoring the modification of the plug-in configuration information and modifying the kernel operating parameters based on the configuration information;

[0016] S104: Compile and install the kernel created in S101 and the plug-in created in S103, turning all the preparatory work into practically usable tools.

[0017] In step S2, the kernel is configured. This step marks the official start of the user's use of the core functions of the present invention, which specifically includes:

[0018] S201: On the Jupyter configuration page, find the configuration tab for the kernel installed in S104 and set the kernel configuration information. This configuration page allows users to customize the configuration of the MPI cluster, including the number of MPI processes and host information.

[0019] S202: The plug-in monitors the configuration information modification and automatically generates the corresponding startup command in the kernel configuration file;

[0020] S203: Select and start the kernel through the Jupyter kernel management interface. At this time, Jupyter Server starts to start the created kernel according to the startup command in the S202 kernel configuration file.

[0021] In step S3, the root node of MPI is operated. The root node usually plays the role of a manager, responsible for allocating tasks and data, and performing special initialization work, including:

[0022] S301: Receive the code to be run from the front-end Jupyter Notebook;

[0023] S302: Broadcast the code to be run to all other non-root nodes to ensure that all nodes can execute the same code;

[0024] S303: The root node executes the code to be run received in S301 in the local environment;

[0025] S304: Collect the execution results of all non-root nodes. Through the MPI communication mechanism, the system can effectively summarize the output and return values from each node;

[0026] S305: Arrange the received operation results and return them to the front end, converting the results of the distributed computing into a form that the user can understand and use.

[0027] In step S4, the non-root node of MPI operates as an executor to complete specific computing tasks, including:

[0028] S401: Listen to and receive the code to be run broadcasted by the root node;

[0029] S402: The non-root node executes the code to be executed received in S401;

[0030] S403: Return the result of this operation to the root node;

[0031] S404: Steps S401 to S403 are repeated, and the non-root node can continuously receive and execute the code from the root node.

[0032] The present invention will repeat S3 to S4 until the front-end Jupyter Notebook chooses to terminate the kernel, then stop all MPI processes, that is, the operation of stopping the process in step S5. At this point, a distributed parallel computing task is completed, specifically including:

[0033] S501: When the user chooses to terminate the kernel, all MPI processes are stopped. Since the running kernel is an MPI cluster, when the root node is shut down, other nodes will also be shut down.

[0034] S502: Update the kernel management status of Jupyter, indicating that the task has been terminated.

[0035] The present invention provides a Jupyter-based distributed parallel computing system, comprising:

[0036] A Jupyter kernel module that supports MPI functionality, used to execute code entered by the user in Jupyter Notebook and return results;

[0037] The kernel configuration management module manages and configures various kernel parameters in the form of Jupyter plug-ins, including the number of MPI processes and host information;

[0038] The task allocation and scheduling module uses MPI distributed parallel computing to distribute the code to different nodes and coordinate the computing process;

[0039] The result aggregation and analysis module uses MPI distributed parallel computing to collect and organize the calculation results returned from each node;

[0040] The user interface module provides easy-to-use console and monitoring tools based on Jupyter.

[0041] The present invention also provides a computer device, including a processor, memory, storage and network interface, characterized in that: the processor executes the Jupyter-based distributed parallel computing method described in any one of steps 1 to 5; the memory provides temporary storage space for running Jupyter Notebook and MPI programs; the storage is used to save Jupyter Notebook, MPI library, Python environment and related configuration files; the network interface is used to connect multiple computer devices to form a distributed computing cluster.

[0042] This paper develops a new MPI-enabled kernel that combines the concepts of Jupyter Server and kernel, enabling interactive analysis and visualization in high-performance computing environments. This innovative technology fuses advanced tools and protocols such as Jupyter and MPI to create a new kernel that can run interactive Python code in an MPI environment and efficiently manage multiple MPI processes.

[0043] This technical solution addresses the difficulty of existing interactive computing environments in directly supporting large-scale distributed parallel computing, bringing new possibilities to fields such as scientific computing and data analysis. It combines the interactivity of Jupyter Notebook with the power of MPI parallel computing, making complex large-scale computing tasks simpler, more efficient, and more intuitive.

[0044] This paper implements a Jupyter-based distributed parallel computing kernel system. By cleverly integrating MPI into Jupyter Notebook, the system is ensured to be correct, simple, and efficient, while also achieving the goal of requiring no additional configuration, no user awareness, and no need to learn other APIs, making it user-friendly. This allows MPI code to be written directly on the Jupyter Notebook page for distributed parallel computing. The specific technical effects are as follows:

[0045] 1) Implemented MPI parallel computing in the Jupyter Notebook environment: This perfectly combines the powerful communication capabilities of MPI with the interactivity of Jupyter Notebook; allowing users to perform stand-alone programming and distributed parallel computing in the same environment.

[0046] 2) Improved computing efficiency: Through multi-node parallel processing, the processing speed of complex computing tasks is significantly improved; it is particularly suitable for large-scale data processing and computationally intensive tasks.

[0047] 3) Simplifies the learning and application process of parallel programming: provides a friendly and intuitive user interface for non-professional parallel programmers; reduces the threshold for learning parallel programming and reduces development costs.

[0048] 4) Enhanced flexibility: Supports dynamic adjustment of computing resources and flexible configuration of the number of MPI processes based on specific tasks; provides an intuitive configuration interface so that users can customize the settings of the MPI cluster.

[0049] 5) Improved user experience: Seamlessly integrated distributed parallel computing is achieved, allowing users to use it without having to understand the underlying details; the interactivity of Jupyter Notebook is retained, making the development and debugging process more intuitive.

[0050] 6) Improved system scalability: supports deployment on multiple machines, can handle ultra-large-scale data and complex computing tasks; allows increasing or decreasing computing nodes according to demand.

[0051] 7) Enhanced visualization capabilities: The visualization output function of Jupyter Notebook is retained in distributed computing results, making it easier for users to intuitively understand and analyze computing results.

[0052] Through these technical effects, the present invention solves the problem that existing interactive computing environments cannot directly support large-scale distributed parallel computing, bringing new possibilities to fields such as scientific computing and data analysis. Based on the design ideas and technical solutions of the present invention, a system for distributed parallel computing using Jupyter Notebook has been implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 It is a schematic diagram of the overall design of the present invention;

[0055] Figure 2 Detailed design flow chart of the present invention;

[0056] Figure 3 This is a schematic diagram of the kernel configuration interface of the present invention;

[0057] Figure 4 Schematic diagram of the system interface of the present invention;

[0058] Figure 5 This is a schematic diagram of the kernel startup process in Example 1 of the present invention;

[0059] Figure 6 It is a schematic diagram of the system design of the present invention;

[0060] Figure 7 Schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0061] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0063] This embodiment aims to integrate MPI parallel computing capabilities into the Jupyter Notebook environment.

[0064] Jupyter is an open source project that provides a powerful scientific computing platform with Jupyter Notebook as its core tool. This platform consists of three main components: Jupyter Server, Jupyter Notebook, and Kernel.

[0065] Jupyter Notebook is an interactive documentation tool that allows users to create documents containing code, visual output, and explanatory text. It provides a friendly interface that allows users to write, execute, and share code in the same environment while recording the entire workflow.

[0066] Jupyter Server is a long-running background process that manages and coordinates the execution of multiple kernels. It acts as a bridge between Jupyter Notebook and other client applications and the kernels, ensuring that all operations are completed safely and efficiently.

[0067] Kernels are core components of the Jupyter ecosystem. Each kernel is an independent process dedicated to executing code in a specific language. When a user enters code in a Jupyter Notebook, the corresponding kernel receives and executes the code, ultimately returning results or output.

[0068] Jupyter Notebook serves as a user interface and maintains a connection to Jupyter Server. When users write and execute code in Notebook, Jupyter Server forwards the request to the corresponding Kernel. JupyterServer acts as an intermediary, managing the starting, stopping, and resource allocation of multiple Kernel. Kernel, as the actual code executor, is responsible for running the user's code in the background. This design enables the Jupyter ecosystem to support multiple programming languages (such as Python) while providing a unified interface and workflow, making scientific computing, data analysis, and visualization simpler and more efficient.

[0069] MPI is a standard parallel programming interface used to coordinate communication and data exchange between multiple computing nodes in a distributed system. The development of the MPI (Message Passing Interface) kernel based on IPython is an innovation that combines the interactive Python environment with the MPI communication mechanism in the field of high-performance computing to support complex interactive programming and visualization requirements. As a basic environment, IPython provides a powerful interactive Python environment. ZMQInteractiveShell is a component in IPython for inter-process communication, which uses the ZeroMQ protocol to implement cross-process communication. The kernel development that supports MPI is a further extension based on IPython and ZMQInteractiveShell, which integrates the MPI parallel programming model into the IPython environment, allowing developers to take advantage of MPI's high-performance communication capabilities while enjoying the interactive programming advantages of IPython.

[0070] Example 1

[0071] like Figure 1 and Figure 2 As shown, this embodiment implements a kernel supporting MPI parallel computing in a Jupyter Notebook environment, including the following steps:

[0072] S1. Kernel creation.

[0073] S1.1. Develop the kernel mpi_kernel based on the IPython project to implement an interactive Python environment that supports MPI functions.

[0074] S1.2. Modify the kernel configuration so that it can be managed by Jupyter. Change the kernel's display name and startup method, and start the kernel using mpiexe.

[0075] S1.3. Develop a Jupyter plugin to configure mpi_kernel and add configuration options to the overall Jupyter configuration.

[0076] S1.3.1. Develop plugin configuration options, including: configuring the number of MPI processes and configuring host information.

[0077] S1.3.2. Modify the configuration information of the monitoring plug-in and modify the kernel operating parameters of mpi_kernel according to the configuration information.

[0078] S1.4. Compile and install the mpi_kernel developed in step S1.1 and the configuration plugin developed in step S1.3. This completes the preparation and turns it into a usable tool.

[0079] S2. After the user opens the Jupyter Notebook page, he or she selects the kernel installed in step S1.4, i.e., mpi_kernel, from the kernel management interface. This step marks the official start of the user's use of the core functions of the present invention.

[0080] S2.1. Open the Jupyter configuration page, find the mpi_kernel configuration tab, and set the mpi_kernel configuration information. This configuration page provides an intuitive and easy-to-use interface that allows users to customize the MPI cluster configuration, such as selecting the number of MPI processes to 6.

[0081] S2.2. The plugin monitors configuration changes and generates the corresponding startup commands in the mpi_kernel configuration file. This process is automated, ensuring that the user-entered configuration is correctly converted into kernel runtime parameters.

[0082] S2.3. Select mpi_kernel in the Jupyter kernel management interface. Jupyter Server will then start the kernel created in step S1 according to the startup command in the mpi_kernel configuration file.

[0083] S3. When mpi_kernel starts, it determines whether its own node is a root node based on its MPI rank. The root node's MPI rank is 0. Root nodes typically act as managers, allocating tasks and data and performing special initialization tasks. Non-root nodes, on the other hand, act as executors, completing specific computational tasks. While these two nodes play different roles in parallel computing, they work together to complete the overall computation. If the node is not a root node, continue with the following steps. If not, skip to step S4.

[0084] S3.1. Receive the code to be run from the front-end Jupyter Notebook. This process ensures that the code entered by the user can be processed and executed correctly.

[0085] S3.2. Broadcast the code to be run to all other non-root nodes. This step is the key to achieving distributed parallel computing, ensuring that all nodes can execute the same code.

[0086] S3.3: The root node executes the code received in step S3.1. This step executes the Python code entered by the user in the local environment.

[0087] S3.4. Collect the running results of all non-root nodes. Through the MPI communication mechanism, the system can effectively summarize the output and return values from each node.

[0088] S3.5. Organize the received running results and return them to the front end. This step converts the results of distributed computing into a form that users can understand and use.

[0089] S4. Non-root nodes are not directly managed by Jupyter and only execute the code to be run broadcasted by the root node.

[0090] S4.1. Listen to and receive the code to be run broadcasted by the root node.

[0091] S4.2: The non-root node executes the code to be executed received in step S4.1. This step ensures that the non-root node can correctly execute the code from the root node.

[0092] S4.3. Return the result of this operation to the root node. This feedback mechanism ensures the synchronization and consistency of the entire cluster.

[0093] S4.4. Repeat steps S4.1 to S4.3. This loop structure enables non-root nodes to continuously receive and execute code from the root node.

[0094] S5. Repeat steps S3 and S4 until the front-end terminates the kernel, stopping all MPI processes. This completes the distributed parallel computing task. The front-end can control kernel shutdowns and restarts. Because the kernel is running as an MPI cluster, shutting down the root node also shuts down all other nodes.

[0095] S5.1. When the user chooses to terminate the kernel, all MPI processes are stopped.

[0096] S5.2. Update the Jupyter kernel management status to indicate that the task has terminated.

[0097] Figure 3 This demonstrates the ability to configure mpi_kernel using Jupyter's configuration interface. This configuration uses six MPI processes and starts all of them on the same host as the Jupyter Server.

[0098] Figure 4 The interface of this system is presented, demonstrating the ability to run general MPI code in Jupyter Notebook using the mpi_kernel. This kernel launches a total of 6 MPI processes. The experimental results verify the correctness of the system.

[0099] Figure 5The Jupyter Server log information is used to show the startup process of the MPI cluster.

[0100] Figure 6 This paper presents a Jupyter-based distributed parallel computing system, including: a Jupyter kernel module that supports MPI functionality, which is used to execute the code entered by the user in the Jupyter Notebook and return the results; a kernel configuration management module that manages and configures various kernel parameters in the form of a Jupyter plug-in, including the number of MPI processes and host information; a task allocation and scheduling module that uses MPI distributed parallel computing to distribute the code to different nodes and coordinate the computing process; a result aggregation and analysis module that uses MPI distributed parallel computing to collect and organize the computing results returned from each node; and a user interface module that provides an easy-to-use console and monitoring tools based on Jupyter.

[0101] Figure 7 This paper presents a computer device comprising a processor, memory, storage, and a network interface. The processor performs MPI parallel computing based on the Jupyter environment; the memory provides temporary storage space for running Jupyter Notebooks and MPI programs; the storage is used to store Jupyter Notebooks, MPI libraries, Python environments, and related configuration files; and the network interface is used to connect multiple computer devices to form a distributed computing cluster.

[0102] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A Jupyter-based distributed parallel computing method, characterized in that: The method includes: S1, creates a Jupyter kernel that supports MPI functions. The kernel is responsible for executing the code entered by the user in the Jupyter Notebook and returning the results; S2, select and configure the kernel in Jupyter Notebook; S3, the root node in MPI responsible for distributing tasks and data to other processes, receiving and broadcasting code, executing code, collecting results and returning them to the front end; S4, the non-root node in MPI that receives instructions and data from the root node and returns the calculation results, executes the broadcast code and returns the results; S5, repeats S3 and S4 until the front end terminates the kernel.

2. A Jupyter-based distributed parallel computing method according to claim 1, characterized in that: Create a Jupyter kernel with MPI support, including: S101: Create a kernel that supports MPI functions based on the IPython project and implement an interactive Python environment that supports MPI functions; S102, modify the kernel configuration information, including changing the kernel's display name and startup method, and using mpiexe to start the kernel so that it can be managed by Jupyter; S103, creating a Jupyter plug-in to configure the kernel, and adding configuration options to the overall configuration of Jupyter, including: developing configuration options for the plug-in, including configuring the number of MPI processes and configuring host information; monitoring the modification of the plug-in configuration information, and modifying the kernel operating parameters based on the configuration information; S104, compile and install the kernel created in S101 and the plug-in created in S103, turning all the preparatory work into a practically usable tool.

3. A Jupyter-based distributed parallel computing method according to claim 1, characterized in that: Select and configure a kernel in Jupyter Notebook in S2, including: S201, on the Jupyter configuration page, find the configuration tab for the kernel installed in S104 and set the kernel configuration information. This configuration page allows users to customize the configuration of the MPI cluster, including the number of MPI processes and host information; S202: The plug-in monitors the configuration information modification and automatically generates the corresponding startup command in the kernel configuration file; S203, select and start the kernel through the kernel management interface of Jupyter, and then Jupyter Server starts to start the created kernel according to the startup command in the kernel configuration file in S202.

4. A Jupyter-based distributed parallel computing method according to claim 1, characterized in that: The root node of MPI in S3 usually plays the role of a manager, responsible for allocating tasks and data, and performing special initialization tasks, including: S301, receiving the code to be run from the front-end Jupyter Notebook; S302, broadcast the code to be run to all other non-root nodes to ensure that all nodes can execute the same code; S303, the root node executes the code to be run received in S301 in the local environment; S304, collect the execution results of all non-root nodes. Through the MPI communication mechanism, the system can effectively summarize the output and return values from each node; S305: sort out the received operation results and return them to the front end, converting the distributed computing results into a form that can be understood and used by users.

5. The Jupyter-based distributed parallel computing method according to claim 1, wherein: The operations of non-root nodes in MPI in S4 are as follows: S401, monitor and receive the code to be run broadcasted by the root node; S402, the non-root node executes the code to be run received in S401; S403, returning the result of this operation to the root node; S404: Repeat steps S401 to S403, and the non-root node can continuously receive and execute the code from the root node.

6. A Jupyter-based distributed parallel computing method according to claims 1 to 5, characterized in that: S3 to S4 are repeated until the front-end Jupyter Notebook chooses to terminate the kernel, which stops all MPI processes. This is the process stop operation in S5. At this point, a distributed parallel computing task ends, including: S501, when the user chooses to terminate the kernel, all MPI processes are stopped. Since the running kernel is an MPI cluster, when the root node is shut down, other nodes will also be shut down; S502: Update the kernel management status of Jupyter to indicate that the task has been terminated.

7. A Jupyter-based distributed parallel computing system, characterized in that: include: A Jupyter kernel module that supports MPI functionality, used to execute code entered by the user in Jupyter Notebook and return results; The kernel configuration management module manages and configures various kernel parameters in the form of Jupyter plug-ins, including the number of MPI processes and host information; The task allocation and scheduling module uses MPI distributed parallel computing to distribute the code to different nodes and coordinate the computing process; The result aggregation and analysis module uses MPI distributed parallel computing to collect and organize the calculation results returned from each node; The user interface module provides easy-to-use console and monitoring tools based on Jupyter.

8. A computer device comprising a processor, a memory, a storage device, and a network interface, characterized in that: The processor executes the Jupyter-based distributed parallel computing method according to any one of claims 1 to 6; the memory provides temporary storage space for running Jupyter Notebook and MPI programs; the storage is used to save Jupyter Notebook, MPI library, Python environment and related configuration files; the network interface is used to connect multiple computer devices to form a distributed computing cluster.

Images