Python-based edge computing engine implementation method and system

Abstract

The embodiment of the present specification provides a kind of based on Python's edge computing engine implementation method and system, wherein, method includes: set local socket as the communication mode between script process and main process, create daemon thread on main process, the daemon thread is used to monitor the script process connected to main process;When creating each edge computing process, create one-way pipe, set the edge computing process end of the one-way pipe as write operation, and the output terminal of edge computing process is redirected to the file descriptor of the one-way pipe of write operation, set the main process end of the pipe as read operation, and read log content;Using file description monitoring function on main process end, all access main process read end file descriptor is monitored, when determining that there is pipe state and / or process state change, log and / or process update operation is carried out.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a method and system for implementing an edge computing engine based on Python. Background Technology

[0002] Edge computing, as a crucial function of edge devices, offers many features that cannot be replaced by the cloud. Because the edge is close to the device, data communication distances are short, and connections are direct, communication issues are generally not a problem. Therefore, some fault detection, early warning, and control functions must be handled at the edge.

[0003] Python, as a simple, easy-to-learn, and easy-to-use scripting language, is very suitable for third-party end-users to customize data processing and logic control. Using Python for edge computing allows users to easily and quickly customize their flexible and specific functional requirements.

[0004] Therefore, how to leverage various Linux mechanisms to ensure data communication, execution scheduling, debugging log output to web terminal users, and flexible script execution strategies between Python script processes on edge devices, thereby guaranteeing that Python achieves its intended purpose on edge devices, is a pressing issue that needs to be addressed. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for implementing an edge computing engine based on Python, aiming to solve the above-mentioned problems in the prior art.

[0006] This invention provides a Python-based edge computing engine implementation method, including:

[0007] Set up a local socket as the communication method between the script process and the main process, and create a daemon thread on the main process. The daemon thread is used to monitor the script processes connected to the main process.

[0008] When creating each edge computing process, a one-way pipe is created, the edge computing process end of the one-way pipe is set to write operation, and the output terminal of the edge computing process is redirected to the file descriptor of the one-way pipe for write operation. The main process end of the pipe is set to read operation to read log content.

[0009] On the main process side, a file descriptor monitoring function is used to monitor all read-end file descriptors connected to the main process. When a change in pipe state and / or process state is detected, log and / or process update operations are performed.

[0010] This invention provides a Python-based edge computing engine implementation system, comprising:

[0011] The configuration module is used to configure the local socket as the communication method between the script process and the main process, and to create a daemon thread on the main process. The daemon thread is used to monitor the script processes connected to the main process.

[0012] A creation module is used to create a one-way pipe when creating each edge computing process, set the edge computing process end of the one-way pipe to write operation, redirect the output terminal of the edge computing process to the file descriptor of the one-way pipe for write operation, set the main process end of the pipe to read operation, and read log content.

[0013] The monitoring module is used on the main process side to monitor all read-end file descriptors connected to the main process using file descriptor monitoring functions, and to perform log and / or process update operations when it is determined that there are changes in pipe status and / or process status.

[0014] This invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the above-described Python-based edge computing engine implementation method.

[0015] This invention also provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, implements the steps of the above-described Python-based edge computing engine implementation method.

[0016] In this embodiment of the invention, firstly, all code writing and operation occur on the main process side, placing no requirements on the edge computing process side; it only needs to start the process and execute the compiled computing code. Secondly, all log information resides in a memory buffer, requiring no writing to any file storage or I / O operations, resulting in faster speed and no damage to local storage media. Thirdly, the log buffer is reusable, flexibly allocating different sizes of space based on the log volume in different scenarios. Fourthly, the main process side can dynamically handle and maintain operations such as adding, ending, and closing new edge computing processes, actively waking them up when data is available and actively suspending them when no data is available. Fifthly, three flexible scheduling and execution strategies can satisfy almost all application scenarios, such as regular computation, periodic processing, and conditional handling of emergency events. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the Python-based edge computing engine implementation method according to an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram illustrating the communication relationship between the script process and the main process in an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of the log operation according to an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of the scheduling strategy logic according to an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of a Python-based edge computing engine implementation system according to an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0025] Method Implementation Examples

[0026] According to an embodiment of the present invention, a method for implementing an edge computing engine based on Python is provided. Figure 1 This is a flowchart of the Python-based edge computing engine implementation method according to an embodiment of the present invention, as shown below. Figure 1 As shown, the Python-based edge computing engine implementation method according to an embodiment of the present invention specifically includes:

[0027] Step 101: Set the local socket as the communication method between the script process and the main process, and create a daemon thread on the main process. The daemon thread is used to monitor the script processes connected to the main process.

[0028] Step 102: When creating each edge computing process, a one-way pipe is created. The edge computing process end of the one-way pipe is set to write operation, and the output terminal of the edge computing process is redirected to the file descriptor of the one-way pipe for write operation. The main process end of the pipe is set to read operation to read log content. Specifically, the process includes the following: The main process starts one or more edge computing processes 1, 2, ..., n. When starting a process, a one-way pipe is created, and the standard output and error output of all edge computing processes are redirected to the one-way pipe. The main process of the pipe is set as the read end. The file descriptor of the one-way pipe is obtained using a common file opening method. The file descriptor monitoring function of select, poll, or epll is used to monitor the file descriptors of all read ends. Once there is content to be read, the main process is woken up by the file descriptor monitoring function to perform a read operation. The log content of each one-way pipe is read and stored in the log buffer pool, where the log buffer pool is stored in a circular buffer mode.

[0029] Step 103: On the main process side, use the file descriptor monitoring function to monitor all read-end file descriptors connected to the main process. When a change in pipe status and / or process status is detected, perform log and / or process update operations. Specifically, this includes the following processing: If the main process detects a request for frontend log content, it returns the logs in the log buffer pool to the frontend for display; if the main process detects that the edge computing process has ended, it deletes the connected unidirectional pipe and shuts down all resources; if the main process detects that the Python process has ended, it monitors and judges new running conditions according to the user-set running strategy. The running strategy specifically includes:

[0030] Startup and execution means that the Python script will run immediately after the device is powered on and started.

[0031] Periodic execution means scheduling a Python script to run at a specified start time and then running it for a specified period.

[0032] Conditional execution determines whether to start the execution of a Python script based on the results of arithmetic and logical operations on specified data collection variables.

[0033] The technical solutions of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0034] The technical solution of this invention is as follows: 1) A local socket is used as the communication method between the script process and the main process. A daemon thread is created on the main process to monitor each script process connected to the main process. 2) When creating each edge computing process, a unidirectional pipe is created. The edge computing process end of the pipe is for write operations, and the output terminal of the edge computing process is redirected to the write operation pipe file descriptor; the main process end of the pipe is for read operations and is responsible for reading log content. 3) System signals and I / O multiplexing serve as the basis for an effective technical solution for logging and multiple execution strategies.

[0035] On the main process side, file descriptor monitoring functions (select, poll, epoll, etc.) are used to monitor all read-end file descriptors connected to the main process. Log and process update operations are performed whenever there are changes in pipe status, process status, etc.

[0036] When the main process detects a request for frontend log content, it returns the logs from the log buffer pool to the frontend for display.

[0037] Once the edge computing process detects that it has finished running, the main process deletes the connected pipes and shuts down all resources.

[0038] When the main process detects that the Python process has terminated, it monitors and determines new running conditions based on the user-defined running policy.

[0039] The strategy for this runtime engine is divided into three types: startup run, which runs the Python script immediately after the device is powered on.

[0040] Periodic execution: Schedules the Python script to run at a specified start time and then executes it within a specified period. Conditional execution: Determines whether to start the Python script based on the results of arithmetic operations and logical operations on specified data collection variables.

[0041] like Figure 2 As shown, multiple script processes establish local socket connections with the main process to exchange data. The scripts retrieve variable values ​​from the main process, process them, and then write them back to the main process. The scripts can set conditions for the main process to monitor; once these conditions are met, the monitoring information is immediately sent to the script processes. The script processes then perform further processing upon receiving the monitoring information.

[0042] like Figure 3As shown, the main process first starts one or more edge computing processes (process 1, process 2, process n, etc.). Simultaneously, a unidirectional pipe is created, and the standard output and error output of all edge computing processes are redirected to the pipe. The main process of the pipe is set as the read end, using a common file opening method to obtain the file descriptor for reading the pipe. File descriptor monitoring functions such as select, poll, or epll are used to monitor all read end file descriptors. Once content is available to read, the monitoring function wakes up the main process to perform the read operation. The log content of each pipe is read and stored in the log buffer pool. The log buffer pool uses a circular buffer method, and its size can be flexibly allocated according to the log system's memory.

[0043] The main process monitors the frontend's log requests. Once a connection request is received, the log information in the log buffer pool is allocated according to a predefined size and sent back to the frontend for processing and display.

[0044] This solution allows one or more edge computing applications running in the background to continuously display logs to the front end. The entire process does not involve physical I / O for reading and writing; it all occurs in memory. When there are no logs or requests, the system remains in a sleep state and does not consume CPU resources.

[0045] like Figure 4 As shown, upon startup, the system first checks if any scripts requiring startup scheduling need to be executed. If so, they are started immediately. Next, it checks if any scripts with a periodic scheduling strategy exist. If so, it immediately starts monitoring the start time. If the time expires, the periodic scheduling script is executed, and after completion, the script process is restarted according to the specified period. Finally, it checks if any scripts with a conditional scheduling strategy exist. If so, it analyzes the startup conditions of each script and begins monitoring whether the calculation conditions are met. Once the startup conditions are met, the corresponding script is executed.

[0046] The following example illustrates this.

[0047] This example uses a Linux system, but it is not limited to Linux; any system that responds to system calls or APIs in the corresponding system can be used. The following is a general description of the implementation process using Linux as an example:

[0048] 1. The Python script and the main program communicate using local sockets.

[0049] 2. For startup, the main process can start the Python script through system, and it will terminate naturally when it finishes running.

[0050] 3. For startup processes with daemons, use popen to start them, and then monitor the file descriptors using select to see if the process has exited.

[0051] 4. For periodic operation scheduling, use timerfd_create to create a runtime timer, and then use select to monitor file descriptors to manage startup time.

[0052] 5. For conditional execution scheduling, the decision on whether to run the specified script is made based on the results of arithmetic and logical operations.

[0053] 6. Log messages are implemented using the popen function, but for fine-grained customization, the above process can be used.

[0054] 7. The main process creates a FIFO as a log data buffer pool, then creates a dedicated thread and uses the select, poll, or epoll functions to monitor the main process's pipe read-end descriptors. When data is detected, the read function is called to read the contents of the pipe and write them to the FIFO buffer pool, modifying the FIFO's write pointer. If the main process detects that a pipe on the compilation / computation side is closed, it removes the corresponding file descriptor from the monitoring list and uses the close function to close the corresponding pipe.

[0055] 8. The main process creates another thread to monitor the socket connections at the front end. Once there is a log read request, it reads the log content from the FIFO and modifies the FIFO's read pointer.

[0056] For other systems, the same process can be achieved by using the corresponding function.

[0057] The beneficial effects of this invention are: First, all code writing and operation are performed on the main process side, requiring no special consideration for the edge computing process side; it only needs to start the process and execute the compiled computing code. Second, all log information is stored in a memory buffer, requiring no writing to any file storage or I / O operations, resulting in faster speed and no damage to local storage media. Third, the log buffer is reusable, flexibly allocating different sizes of space based on the log volume in different scenarios. Fourth, the main process side can dynamically handle and maintain operations such as adding, ending, and closing new edge computing processes, actively waking them up when data is available and actively suspending them when no data is available. Fifth, three flexible scheduling and execution strategies can satisfy almost all application scenarios, such as regular computation, periodic processing, and conditional handling of emergency events.

[0058] System Implementation Examples

[0059] According to embodiments of the present invention, a Python-based edge computing engine implementation system is provided. Figure 5 This is a schematic diagram of a Python-based edge computing engine implementation system according to an embodiment of the present invention, such as... Figure 5 As shown, the Python-based edge computing engine implementation system according to an embodiment of the present invention specifically includes:

[0060] The configuration module 50 is used to configure the local socket as the communication method between the script process and the main process, and to create a daemon thread on the main process. The daemon thread is used to monitor the script processes connected to the main process.

[0061] Module 52 is used to create a unidirectional pipe when creating each edge computing process. The edge computing process end of the unidirectional pipe is set to write operation, and the output terminal of the edge computing process is redirected to the file descriptor of the unidirectional pipe. The main process end of the pipe is set to read operation to read log content. Specifically, the main process starts one or more edge computing processes (process 1, process 2, ..., process n). Simultaneously, a unidirectional pipe is created, and the standard output and error output of all edge computing processes are redirected to the unidirectional pipe. The main process of the pipe is set as the read end. A common file opening method is used to obtain the file descriptor of the unidirectional pipe. The select, poll, or epll file descriptor monitoring functions are used to monitor the file descriptors of all read ends. Once content can be read, the main process is awakened through the file descriptor monitoring function to perform a read operation, reading the log content of each unidirectional pipe and storing it in a log buffer pool. The log buffer pool uses a circular buffer method for storage.

[0062] The monitoring module 54 is used to monitor all read-end file descriptors connected to the main process using file descriptor monitoring functions on the main process side. When a change in pipe status and / or process status is detected, log and / or process update operations are performed. Specifically, it is used for: when the main process detects a request for front-end log content, returning the logs from the log buffer pool to the front-end for display; when the main process detects the edge computing process has ended, deleting the connected unidirectional pipe and shutting down all resources; when the main process detects the Python process has ended, monitoring and judging new running conditions according to the user-defined running strategy. The running strategy specifically includes:

[0063] Startup and execution means that the Python script will run immediately after the device is powered on and started.

[0064] Periodic execution means scheduling a Python script to run at a specified start time and then running it for a specified period.

[0065] Conditional execution determines whether to start the execution of a Python script based on the results of arithmetic and logical operations on specified data collection variables.

[0066] The embodiments of the present invention are system embodiments corresponding to the above method embodiments. The specific operation of each module can be understood by referring to the description of the method embodiments, and will not be repeated here.

[0067] Device Example 1

[0068] This invention provides an electronic device, such as... Figure 6 As shown, it includes: a memory 60, a processor 62, and a computer program stored in the memory 60 and executable on the processor 62, wherein the computer program, when executed by the processor 62, performs the steps as described in the method embodiment.

[0069] Device Example 2

[0070] This invention provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor 62, performs the steps described in the method embodiment.

[0071] The computer-readable storage media described in this embodiment include, but are not limited to, ROM, RAM, disk, or optical disk.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for implementing an edge computing engine based on Python, characterized in that, include: Set up a local socket as the communication method between the script process and the main process, and create a daemon thread on the main process. The daemon thread is used to monitor the script processes connected to the main process. When creating each edge computing process, a one-way pipe is created, the edge computing process end of the one-way pipe is set to write operation, and the output terminal of the edge computing process is redirected to the file descriptor of the one-way pipe for write operation. The main process end of the pipe is set to read operation to read log content; specifically including: The main process starts one or more edge computing processes (process 1, process 2, ..., process n). Simultaneously, a one-way pipe is created, and the standard output and error output of all edge computing processes are redirected to this one-way pipe. The main process of the pipe is set as the read end. A common file opening method is used to obtain the file descriptor of the one-way pipe. The select, poll, or epll file descriptor monitoring functions are used to monitor the file descriptors of all read ends. Once content is available to read, the main process is awakened through the file descriptor monitoring function to perform the read operation. The log content of each one-way pipe is read and stored in a log buffer pool, which uses a circular buffer method for storage. On the main process side, a file descriptor monitoring function is used to monitor all read-end file descriptors connected to the main process. When a change in pipe state and / or process state is detected, log and / or process update operations are performed.

2. The method according to claim 1, characterized in that, On the main process side, a file descriptor monitoring function is used to monitor all read-end file descriptors connected to the main process. When a change in pipe state and / or process state is detected, log and / or process update operations are performed, specifically including: When the main process detects a request for frontend log content, it returns the logs from the log buffer pool to the frontend for display. When the main process detects that the edge computing process has finished running, it deletes the connected unidirectional pipe and shuts down all resources. When the main process detects that the Python process has finished running, it monitors and judges new running conditions according to the user-set running policy.

3. The method according to claim 2, characterized in that, The specific operational strategy includes: Startup and execution means that the Python script will run immediately after the device is powered on and started. Periodic execution means scheduling a Python script to run at a specified start time and then running it for a specified period. Conditional execution determines whether to start the execution of a Python script based on the results of arithmetic and logical operations on specified data collection variables.

4. A Python-based edge computing engine implementation system, characterized in that, include: The configuration module is used to configure the local socket as the communication method between the script process and the main process, and to create a daemon thread on the main process. The daemon thread is used to monitor the script processes connected to the main process. A creation module is used to create a unidirectional pipe when creating each edge computing process. The edge computing process end of the unidirectional pipe is set to write operation, and the output terminal of the edge computing process is redirected to the file descriptor of the write-operated unidirectional pipe. The main process end of the pipe is set to read operation to read log content. Specifically, it is used for: The main process starts one or more edge computing processes (process 1, process 2, ..., process n). Simultaneously, a one-way pipe is created, and the standard output and error output of all edge computing processes are redirected to this one-way pipe. The main process of the pipe is set as the read end. A common file opening method is used to obtain the file descriptor of the one-way pipe. The select, poll, or epll file descriptor monitoring functions are used to monitor the file descriptors of all read ends. Once content is available to read, the main process is awakened through the file descriptor monitoring function to perform the read operation. The log content of each one-way pipe is read and stored in a log buffer pool, which uses a circular buffer method for storage. The monitoring module is used on the main process side to monitor all read-end file descriptors connected to the main process using file descriptor monitoring functions, and to perform log and / or process update operations when it is determined that there are changes in pipe status and / or process status.

5. The system according to claim 4, characterized in that, The monitoring module is specifically used for: When the main process detects a request for frontend log content, it returns the logs from the log buffer pool to the frontend for display. When the main process detects that the edge computing process has finished running, it deletes the connected unidirectional pipe and shuts down all resources. When the main process detects that the Python process has finished running, it monitors and judges new running conditions according to the user-set running policy.

6. The system according to claim 4, characterized in that, The specific operational strategies include: Startup and execution means that the Python script will run immediately after the device is powered on and started. Periodic execution means scheduling a Python script to run at a specified start time and then running it for a specified period. Conditional execution determines whether to start the execution of a Python script based on the results of arithmetic and logical operations on specified data collection variables.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the Python-based edge computing engine implementation method as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an information transmission implementation program, which, when executed by a processor, implements the steps of the Python-based edge computing engine implementation method as described in any one of claims 1 to 3.

Images