Code processing method, computing device and computer program product

Abstract

The embodiment of the invention provides a code processing method, computing equipment and a computer program product, and the code processing method comprises the steps that a target code and a target code processing unit corresponding to the target code are determined; the target code processing unit is used for determining a target code processing subunit for processing the target code from a plurality of code processing subunits, the code processing subunits are generated according to the target code processing unit, and each code processing subunit is provided with a corresponding code interpreter; using the target code processing unit to send the target code to the target code processing subunit; processing the target code by utilizing the target code processing subunit and a target code interpreter corresponding to the target code processing subunit to obtain a code processing result corresponding to the target code; therefore, the limitation of a global interpreter lock is broken through, and the code execution efficiency is improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular, to a code processing method; one or more embodiments of this specification also relate to a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In current software development practices, cross-language programming and integration is a common and valuable task; for example, Python has been widely used with its rich libraries and simple syntax, especially in fields such as data analysis, machine learning, and web development.

[0003] However, in the process of executing the code, the Global Interpreter Lock (GIL) mechanism simplifies memory management, but also limits the parallel execution capability of the program on multi-core processors. For example, Python's Global Interpreter Lock (GIL) limits its execution efficiency in a multi-threaded environment, because GIL ensures that only one thread among multiple threads contained in a process can execute Python bytecode at the same time, which severely limits the parallel processing capability.

[0004] Because the interpreter using GIL only allows multiple threads contained in a process and only executes one thread at the same time, if multiple threads want to execute a certain section of code at the same time, they have to wait in line, resulting in inefficient execution performance. Summary of the invention In view of this, an embodiment of this specification provides a code processing method. One or more embodiments of this specification also relate to a code processing device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a code processing method is provided, including: Determine the target code and the target code processing unit corresponding to the target code; Determining, by using the target code processing unit, a target code processing subunit for processing the target code from a plurality of code processing subunits, wherein the code processing subunit is generated according to the target code processing unit and each code processing subunit has a corresponding code interpreter; Using the object code processing unit, sending the object code to the object code processing subunit; The target code is processed by using the target code processing subunit and the target code interpreter corresponding to the target code processing subunit to obtain a code processing result corresponding to the target code.

[0006] According to a second aspect of an embodiment of this specification, a code processing device is provided, including: A first unit determination module is configured to determine a target code and a target code processing unit corresponding to the target code; A second unit determination module is configured to use the target code processing unit to determine a target code processing subunit that processes the target code from a plurality of code processing subunits, wherein the code processing subunit is generated according to the target code processing unit and each code processing subunit has a corresponding code interpreter; A code sending module, configured to send the target code to the target code processing sub-unit by using the target code processing unit; The code processing module is configured to process the target code using the target code processing subunit and the target code interpreter corresponding to the target code processing subunit to obtain a code processing result corresponding to the target code.

[0007] According to a third aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above-mentioned code processing method are implemented.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and the steps of the above-mentioned code processing method are implemented when the computer program / instruction is executed by a processor.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which implements the steps of the above-mentioned code processing method when executed by a processor.

[0010] The code processing method provided in one or more embodiments of the present specification can generate multiple code processing sub-units according to the target code processing unit, and each code processing sub-unit maintains a corresponding code interpreter; therefore, in the process of executing the code, the target code is sent to the target code processing sub-unit that processes the target code by using the target code processing unit, and the target code processing sub-unit and the target code interpreter maintained by it are used to execute the target code in parallel to obtain the code processing result corresponding to the target code; thereby, by maintaining a code interpreter for each code processing sub-unit, the limitation of the global interpreter lock is broken, so that multiple code processing sub-units can execute code in parallel at the same time, thereby improving the efficiency of code execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is an execution schematic diagram using a global interpreter lock provided by an embodiment of this specification; Figure 2 It is an application diagram of a code processing method provided by an embodiment of this specification; Figure 3 is a flow chart of a code processing method provided by an embodiment of this specification; Figure 4 It is a schematic diagram of executing code parsing using an abstract syntax tree in a code processing method provided by an embodiment of this specification; Figure 5 is a schematic diagram of a tree structure in a code processing method provided by an embodiment of this specification; Figure 6 is a schematic diagram of another tree structure in a code processing method provided by an embodiment of this specification; Figure 7 It is a schematic diagram of a process exit in a code processing method provided by an embodiment of this specification; Figure 8 is a schematic diagram of the structure of a code processing device provided by an embodiment of this specification; Fig. 9 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0012] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0013] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0015] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0016] First, the terms involved in one or more embodiments of this specification are explained.

[0017] Java: is a widely used high-level, object-oriented programming language. The main feature of Java is Write Once, Run Anywhere (WORA), which is due to the architecture of the Java Virtual Machine (JVM).

[0018] Go: Also known as Golang, it is a statically typed, compiled language with features such as garbage collection and built-in concurrency support; its design goal is to improve program writing and running efficiency.

[0019] Python: is a high-level programming language known for its clear syntax and code readability; Python supports multiple programming paradigms, including object-oriented, imperative, functional, and procedural programming.

[0020] C++: is a high-level programming language that supports multi-paradigm programming (including object-oriented and generic programming); it is a superset of C language with stronger type checking and support for object-oriented features such as classes (classes).

[0021] GIL (Global Interpreter Lock): refers to the global interpreter lock. GIL is a mechanism in Python that is used to prevent multiple native threads from executing Python bytecode simultaneously at the interpreter level, thereby simplifying memory management, but also limiting the parallel execution capabilities of the program on multi-core processors.

[0022] Thread: It is the smallest unit that the operating system can schedule operations on. It is contained in the process and is the actual operation unit in the process. Threads are relatively lightweight and share process resources.

[0023] Process: It is an operation of a program in a computer on a certain data set. It is the basic unit for resource allocation and scheduling in the system and the foundation of the operating system structure.

[0024] Multithreading: refers to the implementation of multiple threads of execution from a software or hardware perspective, so that multiple tasks can run simultaneously in a single process.

[0025] Multi-process: refers to running multiple processes simultaneously in an application. Each process has its own independent code and data space and is isolated from each other.

[0026] Multi-thread pool: a resource pool that stores multiple initialized threads. It can effectively manage and allocate thread resources to improve program performance and resource utilization.

[0027] Multi-process pool: Similar to the multi-thread pool, but it manages process resources; the multi-process pool can be used to perform parallel computing and fully utilize the computing power of multi-core processors.

[0028] JNI (Java Native Interface): is the interface standard for Java to call native (C language / C++) code, allowing Java code to interact with applications or libraries written in other languages.

[0029] AST (Abstract Syntax Tree): refers to the abstract syntax tree. AST is a tree representation of the abstract syntax structure of the source code, which is used for syntax analysis and other processing.

[0030] Dynamic library file: A file that contains code and data that can be used by multiple programs at the same time. It is usually a dynamic link library (DLL) file on Windows operating systems and a .so file on Unix-like operating systems.

[0031] SWIG (Simplified Wrapper and Interface Generator): is a software development tool used to connect C and C++ programs to various high-level programming languages ​​and generate connection code, thereby allowing function calls between programs written in different programming languages.

[0032] web (World Wide Web): the global wide area network, also known as the World Wide Web, is a global, dynamically interactive, cross-platform distributed graphic information system based on hypertext and HTTP (Hypertext Transfer Protocol).

[0033] Jython: is a complete language, not a Java translator or just a Python compiler, it is a complete implementation of the Python language in Java. Jython also has many module libraries inherited from CPython.

[0034] JEP (Java expression parser): refers to the Java expression analyzer. Jep is a Java library used to convert and calculate mathematical expressions.

[0035] JPype: is a tool that allows Python code to easily call Java code.

[0036] ‌GraalVM: is a high-performance virtual machine based on the ‌JVM, capable of running ‌Java and other languages ​​such as ‌JavaScript with excellent performance.

[0037] API (Application Programming Interface): refers to the application programming interface.

[0038] In current software development practices, cross-language programming and integration is a common and valuable task; however, during code execution, the Global Interpreter Lock (GIL) mechanism is used, which simplifies memory management but also limits the parallel execution capability of the program on multi-core processors.

[0039] For example, in current software development practices, cross-language programming and integration is a common and valuable task; Python has been widely used with its rich libraries and simple syntax, especially in fields such as data analysis, machine learning, and web development. However, Python's global interpreter lock (GIL) limits its execution efficiency in a multi-threaded environment, because the GIL ensures that only one thread can execute Python bytecode at the same time, which severely limits parallel processing capabilities. Although there are a variety of tools and methods that support embedding or calling Python code in languages ​​​​such as Java and C++, most of these methods are still restricted by GIL and cannot fully utilize the parallel computing capabilities of modern multi-core processors. This limitation not only applies to the integration of Java and Python, but also affects the integration efficiency and performance of many other programming languages ​​​​with Python; for details, please refer to Figure 1 .

[0040] Figure 1 This is an execution diagram using a global interpreter lock provided by an embodiment of this specification; due to Python's GIL (Global Interpreter Lock) restrictions, when using C++ to integrate the Python interpreter, only one thread is executing at any time. Even on a multi-core processor, the interpreter using GIL only allows one thread to execute at the same time. That is to say, if multiple threads want to execute a certain section of Python code at the same time, they have to wait in line, resulting in inefficient execution performance. Based on Figure 1 It can be seen that the main thread can start the Python interpreter and wait to receive Python code; if there is Python code input, the Python code is executed, the execution result is obtained, and then waits again; if there is no Python code input yet, it continues to wait; when the main thread receives an exit instruction, it ends the code processing and exits.

[0041] Based on the above, we can see that because Python's GIL lock is global, it will not work even if multiple threads are started and each thread maintains a Python interpreter.

[0042] To solve the above problems, this manual provides the following two solutions: The first solution is to use Jython. However, since Jython only supports Python 2 syntax, and Python 2 was abandoned several years ago, and the Jython version that supports Python 3 has been slow to come out, it cannot effectively solve the above problems. In addition, this solution starts a Java gateway server to communicate with Python, but communication over the network may result in lower performance than using JNI directly.

[0043] The second solution is to use Jep and JPype. Both solutions embed the Python interpreter directly into the Java application through JNI. However, the disadvantages are: complex configuration; and only support Java, so the applicability is poor.

[0044] The third solution is to use GraalVM. The GraalVM in this solution is itself a virtual machine that allows Python to run on this virtual machine. However, the disadvantage of this solution is that it is not conducive to use because it requires other software to do a lot of adaptation work.

[0045] Based on this, in this specification, a code processing method is provided. This specification also relates to a code processing device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0046] See also Figure 2 , Figure 2 A schematic diagram of an application of a code processing method provided according to an embodiment of the present specification is shown. Figure 2 It can be seen that the main process forks multiple child processes, each of which maintains a Python interpreter; in the process of executing the code, each child process starts the Python interpreter and waits to receive the Python code; the main process sends the Python code to be executed to the child process; when the child process is sure that the Python code has been received, it executes the Python code and obtains the execution result; after the child process is executed, the execution result is sent to the main process, which is then output by the main process, thus breaking through the limitation of the global interpreter lock, so that multiple child processes can execute code in parallel at the same time, improving the efficiency of code execution.

[0047] See also Figure 3 , Figure 3 A flowchart of a code processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0048] Step 302: Determine the target code and the target code processing unit corresponding to the target code.

[0049] Among them, the target code can be understood as the code written during the software development process.

[0050] The code processing unit can be understood as a unit used to process the code. For example, the code processing unit can be a process, a virtual machine, a script engine, a container, etc.; there are multiple code processing units, and the target code processing unit is a code processing unit among the multiple code processing units that processes the target code.

[0051] Among them, determining the target code processing unit corresponding to the target code can be understood as determining an idle code processing unit among multiple code processing units as the target code processing unit corresponding to the target code; that is, determining an idle code processing unit from multiple code processing units, and using it as the target code processing unit for processing the target code; the idle code processing unit can be understood as a code processing unit that has no executed code but is in a running state, or a code processing unit with a low load.

[0052] Among them, determining the target code processing unit corresponding to the target code can also be understood as creating or generating a corresponding target code processing unit for the target code; when multiple code processing units are busy (i.e., the load is high), a target code processing unit for processing the target code can be recreated or generated; for example, when the code processing unit is a process, a target process can be allocated to the target code from the process resources in the multi-process pool for executing the target code.

[0053] Step 304: Determine, using the target code processing unit, a target code processing subunit for processing the target code from a plurality of code processing subunits, wherein the code processing subunit is generated according to the target code processing unit, and each code processing subunit has a corresponding code interpreter.

[0054] The code processing subunit can be understood as a unit used to process the target code. The code processing subunit can be a process, a virtual machine, a script engine, a container, etc.; there can be multiple code processing subunits, and the target code processing subunit can be understood as a code processing subunit that processes the target code.

[0055] It should be noted that the code processing sub-unit is generated according to the target code processing unit, which can be understood as the code processing sub-unit is generated by forking the target code processing unit; for example, the code processing unit is a main process, and the code processing unit is a child process, and multiple child processes can be forked based on the main process.

[0056] That is to say, in one or more embodiments provided in this specification, the target code processing unit is a main process, the code processing sub-unit is a child process, the child process includes multiple threads, and the child process is generated by replicating the main process, thereby overcoming the problem of global interpreter lock (GIL) limitation based on the child process obtained by replicating.

[0057] In one or more embodiments provided in this specification, the replication operation can be implemented by calling an API interface in the operating system, and the API is used to implement the process replication operation.

[0058] The code interpreter can be understood as a component that reads, executes and analyzes the target code. The code interpreter can be a Python interpreter, which reads, parses and executes Python code.

[0059] Each code processing sub-unit has a corresponding code interpreter, which can be understood as maintaining a Python interpreter for each sub-process.

[0060] In one or more embodiments provided in this specification, determining a target code processing subunit for processing the target code from a plurality of code processing subunits includes: Determine an idle code processing subunit among the plurality of code processing subunits as a target code processing subunit corresponding to the target code; that is, determine an idle code processing subunit from the plurality of code processing subunits and use it as a target code processing subunit for processing the target code; Alternatively, a corresponding target code processing subunit is created or generated for the target code; that is, when multiple code processing subunits are busy (i.e., the load is high), a target code processing subunit for processing the target code can be recreated or generated; for example, when the code processing subunit is a subprocess, a target subprocess can be allocated to the target code from the subprocess resources in the multi-process pool to execute the target code.

[0061] The idle code processing subunit may be understood as a code processing subunit that is not executing code but is in a running state, or a code processing subunit with a low load.

[0062] Step 306: Utilize the target code processing unit to send the target code to the target code processing subunit.

[0063] Continuing with the above example, the code processing method provided in this manual is that the main process forks multiple sub-processes, and each sub-process maintains a Python interpreter; during the code execution process, the main process sends the Python code to be executed to the sub-process; this breaks through the limitation of Python's GIL on multi-threading and greatly improves the efficiency of executing Python code.

[0064] Step 308: Utilize the target code processing subunit and the target code interpreter corresponding to the target code processing subunit to process the target code and obtain a code processing result corresponding to the target code.

[0065] The code processing result can be understood as the result obtained after executing the target code. For example, when the target code is an expression, the code processing result can be understood as the numerical value obtained by executing the expression, or the return value obtained by executing the expression.

[0066] In one or more embodiments provided in this specification, the target code processing subunit and the target code interpreter corresponding to the target code processing subunit are used to process the target code to obtain a code processing result corresponding to the target code, including: Utilizing the target code processing subunit, receiving the target code, and calling the target code interpreter, parsing the target code according to the abstract syntax tree, and determining a plurality of code statements in the target code; A target code statement containing an expression is determined from the multiple code statements, and the target code statement is executed to obtain a code processing result corresponding to the target code.

[0067] It should be noted that when using the code method to integrate the Python interpreter API, the core will call the execute and evaluate methods for the execution of Python code; both methods can execute Python code, but the difference is: The evaluate function is used to evaluate a valid Python expression stored in a string (i.e., the target code statement containing the expression) and return the result of the expression (i.e., the result of code processing); it can only be used for a single expression, which means that evaluate can perform operations such as addition, subtraction, multiplication, division, function calls, or any other operations that return values; the following is an example of using evaluate: result = evaluate("3 + 5") print(result)# Output 8 The execute function executes Python code stored in a string or other code object (i.e., code statements to be executed that do not contain expressions); unlike evaluate, execute can execute complex Python code (i.e., code statements to be executed that do not contain expressions), including multiple lines of code, loops, if statements, function definitions, etc., but it does not return any results. The following is an example of using execute: code = """ for i in range(5): print(i) """ execute(code) Because of the design of the Python interpreter API, developers need to pay attention to whether the Python code being executed is an expression or a non-expression; if it is an expression, they often need to get the execution result, but if it is not an expression, they don't need to. This is very unfriendly to developers.

[0068] Therefore, the code processing method provided in this specification analyzes the Python code to be executed through an abstract syntax tree, and based on the abstract syntax tree, obtains which expressions can be evaluated and which do not need to be evaluated in the Python interpreter integrated device, thereby accurately selecting the optimal execution method to execute the target code, thereby improving the code execution efficiency; see Figure 4 , Figure 4 is a schematic diagram of executing code parsing using an abstract syntax tree in a code processing method provided in an embodiment of the present specification, based on Figure 4 It can be seen that for Python code, the Python interpreter of the child process parses the code through the abstract syntax tree and divides the code into multiple statements, namely Figure 4 “Python statement 1 import sys”, “Python statement 2 a =sys.path”, “Python statement 3 print(a)”.

[0069] Then, the Python interpreter determines whether each Python statement can be evaluated; if so, it evaluates (Python code) and obtains the result; if not, it evaluates (Python code).

[0070] In one or more embodiments provided in this specification, calling the target code interpreter, parsing the target code according to the abstract syntax tree, and determining multiple code statements in the target code include: Calling the target code interpreter, parsing the target code according to the abstract syntax tree, and obtaining a tree structure corresponding to the target code, wherein nodes in the tree structure are generated according to code statements; Based on the tree structure, multiple code statements in the target code are determined.

[0071] Continuing with the above example, in the process of using the abstract syntax tree to determine which specific statements in the target code can be evaluated, the core is based on the parsing result of the abstract syntax tree; for example, the following code: def foo(x):return x + 1 The above code is parsed into a tree structure after the abstract syntax tree is parsed. The tree structure can be found in Figure 5 , Figure 5 This is a schematic diagram of a tree structure in a code processing method provided in an embodiment of this specification. From this abstract syntax tree, we can see that there is a function definition node: FunctionDef, and then FunctionDef also includes: Return (return value statement), BinOp (operation statement), and FunctionDef also includes Arguments (parameters). Based on the abstract syntax tree, it can be determined that the statements that can be evaluated can be: A relatively simple expression, sample code: x = 10, or result = x * 2 Lambda expression, sample code: lam = lambda y: y + 1 Dictionary (dict) expression, example code: dict_comp = {k: k * k for k in range(5)} Set expression, sample code: set_comp = {s for s in range(7) if s % 2 == 0} Generator expression, example code: gen_exp = (g * g for g in range(10)) List expression, example code: list_comp = [l ** 2 for l in range(10)] That is to say, if based on the analysis of the abstract syntax tree, if a node is found to have the above expression, it is considered to be evaluable, such as Figure 6 As shown, Figure 6 is a schematic diagram of another tree structure in a code processing method provided in an embodiment of this specification, Figure 6 The node “Expr:os.listdir('.')” filled with black may be the code statement being evaluated.

[0072] Based on the above content, it can be known that determining the target code statement containing the expression from the multiple code statements includes: Based on the tree result corresponding to the target code, determining the target code statement containing the expression from the multiple code statements; The step of determining, from the plurality of code statements, a to-be-executed code statement that does not contain an expression comprises: Based on the tree result corresponding to the target code, a target code statement that does not contain an expression is determined from the multiple code statements.

[0073] In one or more embodiments provided in this specification, the Python interpreter integrated device can be understood as a device for implementing the code processing method provided in one or more embodiments of this specification, and the Python interpreter integrated device can be obtained by writing in a programming language.

[0074] In one or more embodiments provided in this specification, the calling of the target code interpreter, parsing the target code according to the abstract syntax tree, and determining multiple code statements in the target code further includes: A to-be-executed code statement that does not contain an expression is determined from the multiple code statements, and the to-be-executed code statement is executed.

[0075] Using the above example, based on the abstract syntax tree, the Python interpreter integrated device can determine which expressions can be evaluated and which do not need to be evaluated; for code statements that do not need to be evaluated, the Python code is executed, but no code processing results need to be returned. This enables accurate selection of the optimal execution method to execute the target code, improving code execution efficiency.

[0076] In one of the multiple embodiments provided in this specification, after the target code is processed by the target code processing subunit and the target code interpreter corresponding to the target code processing subunit to obtain the code processing result corresponding to the target code, the method further includes: Sending the code processing result to the target code processing unit; The target code processing unit is used to send the code processing result to a result acquisition unit.

[0077] Among them, the result acquisition unit can be understood as a unit that needs to obtain the code processing result; the result acquisition unit can be a code providing unit that provides the target code through a function; for example, the code processing result can be an operating system, other programming languages, scripts, applications, etc.

[0078] Continuing with the above example, the main process sends the Python code to be executed to the child process. After the child process is executed, the main process returns, thus breaking through the limitation of Python's GIL on multi-threading and greatly improving the efficiency of executing Python code.

[0079] In one or more embodiments provided in this specification, the operation of the code processing subunit needs to be based on the control of the target code processing unit. The code processing result of the code processing subunit will be sent to the target code processing unit and output. The target code processed by the code processing subunit is sent by the target code processing unit. At the same time, whether the target code processing subunit exits is also based on the control of the target code processing unit. The specific implementation method is as follows.

[0080] The method further comprises: When receiving an exit instruction for the target code processing unit, exiting the plurality of code processing sub-units by sending the exit instruction to the plurality of code processing sub-units using the target code processing unit; When it is determined that the exit of the plurality of code processing sub-units is completed, the target code processing unit is exited.

[0081] Among them, exit can be understood as the behavior of a running code processing subunit or code processing unit ending its own operation when completing its task or encountering a certain termination condition. For example, when the code processing subunit is a subprocess and the code processing unit is a main process, the exit can be understood as "process exit", which refers to the behavior of a running process (i.e., an instance of a program) ending its own operation when completing its task or encountering a certain termination condition.

[0082] Using the above example, Figure 7 is a schematic diagram of a process exit in a code processing method provided in an embodiment of this specification, based on Figure 7It can be seen that when the main process receives the "exit" instruction, it also sends an "exit" instruction to all child processes, thereby notifying the child processes to perform the "exit" operation, and all child processes execute the exit operation based on the exit instruction; the main process will also determine and wait whether all child processes have exited. When all child processes have exited, the main process exits.

[0083] In the above embodiment, the target code processing unit controls multiple code processing sub-units, thereby achieving efficient management of the multiple code processing sub-units and avoiding confusion in process management of the operating system due to replicating multiple code processing sub-units.

[0084] In one or more embodiments provided in this specification, the code processing method supports the integration of code interpreters (i.e., Python interpreters) in multiple programming languages; specifically, since the Python interpreter itself is developed in the C++ language, it has better support for C++, that is, C++ can be relatively easy to integrate the Python interpreter. However, the integration of other programming languages ​​into the Python interpreter is not simple. From the comparison above, it can be seen that the current implementations on the market are only for a certain programming language and cannot be implemented for multiple languages. In the code processing method provided in this specification, innovations have been made to address this problem, allowing multiple programming languages ​​to integrate the Python interpreter. The specific implementation process is as follows: The determining of the target code comprises: Receiving a target code sent by a code providing unit, wherein the target code is sent by the code providing unit by calling a code providing component, and the code providing component includes a function and an interface; Based on the target code, dynamic library files corresponding to the target code processing unit and the multiple code processing sub-units are loaded, and the target code processing unit and the multiple code processing sub-units are controlled to process the target code.

[0085] The code providing unit may be understood as a unit that provides target code to the target code processing unit. For example, the code providing unit may be an operating system, other programming languages, scripts, applications, and the like.

[0086] The code providing component can be understood as a component that can help the code providing unit provide the target code. The code providing component includes functions and interfaces. For example, the function can be the ExecuteCode(PythonCode) function, which accepts Python code, executes Python code, and returns the execution result; for example, the interface can be a Java interface code or a Go interface code.

[0087] Taking the application of the code processing method provided in this specification in the scenario of Python interpreter integration as an example, the code processing method is explained. Among them, the code processing unit is the main process, and the code processing sub-unit is the sub-process. Based on this, the code processing method provided in this specification provides a high-performance, easy-to-use, and multi-programming language Python interpreter integration device. The basic code of the device is developed in C++. In this C++ developed code, the core provides a function named ExecuteCode(PythonCode), which accepts Python code, executes Python code, and returns after obtaining the execution result.

[0088] In addition, through the glue language SWIG, the above C++ code is converted into codes of programming languages ​​such as Java interface code, Go interface code, and corresponding dynamic library files; while other programming languages ​​(such as Java) only need to load the above dynamic library files and call the above Java interface to complete the language's parsing of the Python interpreter.

[0089] In the above embodiment, by loading the dynamic library and calling the code providing component, the target code processing unit and multiple code processing sub-units can be controlled to process the target code, thereby achieving flexible processing of the target code and improving the applicability of the method.

[0090] Taking the application of the code processing method provided in this specification in supporting the integration of multiple programming languages ​​with the Python interpreter as an example, the code processing method is further described. Among them, the processing process of a code processing method provided in an embodiment of this specification specifically includes the following three aspects.

[0091] The first aspect: concurrent calls to the embedded Python interpreter are realized through the subprocess pool.

[0092] Due to Python's GIL (Global Interpreter Lock), when using C++ to integrate the Python interpreter, only one thread is executed at any time. Even on a multi-core processor, the interpreter using GIL only allows one thread to execute at the same time. In other words, if multiple threads want to execute a piece of Python code at the same time, they have to wait in line, resulting in inefficient execution performance.

[0093] And because Python's GIL lock is global, it is not feasible to start multiple threads and maintain a Python interpreter for each thread. Based on this, the code processing method provided in this manual proposes a method to implement concurrent calls to the embedded Python interpreter through a subprocess pool. The specific implementation method is: First, the main process forks multiple subprocesses, and each subprocess maintains a Python interpreter; Secondly, the main process sends the Python code to be executed to the subprocess, and the subprocess and the Python interpreter maintained by the subprocess execute the Python code to obtain the code execution result; Finally, when the subprocess is executed, the code execution results are sent to the main process, and then output by the main process. This breaks through the limitations of Python's GIL on multi-threading and greatly improves the efficiency of executing Python code.

[0094] In addition, it should be noted that when the main process receives the "exit" command, it will also notify all child processes to perform the "exit" operation. The main process will only exit when all child processes have exited.

[0095] The second aspect: The API design combines the two code execution methods into one, simplifying the usage difficulty for users.

[0096] When using the code method to integrate the Python interpreter API, the core will call the execute and evaluate methods for the execution of Python code. Both methods can execute Python code, but the differences are: The evaluate function is used to evaluate a valid Python expression stored in a string and return the result of the expression. It can only be used for a single expression. This means that you can perform operations such as addition, subtraction, multiplication, division, calling functions, or any other operations that return values ​​through evaluate.

[0097] The execute() function executes Python code stored in a string or other code object. Unlike evaluate(), execute() can execute complex Python code, including multiple lines of code, loops, if statements, defined functions, etc., but it does not return any results.

[0098] Because of the design of the Python interpreter API, developers need to pay attention to whether the Python code being executed is an expression or a non-expression; if it is an expression, they often need to get the execution result, but if it is not an expression, they don't need to. This is very unfriendly to developers.

[0099] Therefore, the Python interpreter integrated device performs an abstract syntax tree analysis on the Python code to be executed, and based on the abstract syntax tree, obtains in the Python interpreter integrated device which expressions can be evaluated and which do not need to be evaluated.

[0100] The core of determining which statements can be evaluated is based on the parsing results of the abstract syntax tree; for example, the following code: def foo(x): return x + 1 It is a tree structure obtained by parsing the abstract syntax tree. Figure 5 ; Through this abstract syntax tree, we can see that there is a function definition node: FunctionDef, and FunctionDef also includes: Return (return value statement), BinOp (operation statement), and FunctionDef also includes Arguments (parameters).

[0101] Based on the abstract syntax tree, we can determine the statements that can be evaluated; that is, if based on the analysis of the abstract syntax tree, a node is found to have the above expression, then the code statement is considered to be able to be evaluated, otherwise, the code statement is considered to be able to be executed.

[0102] The third aspect: support for multiple programming languages ​​​​with built-in Python interpreter.

[0103] Since the Python interpreter itself is developed in C++, it has better support for C++, that is, C++ can be easily integrated with the Python interpreter. However, it is not easy to integrate other programming languages ​​with the Python interpreter. The current implementations on the market are only for a certain programming language and cannot be implemented for multiple languages.

[0104] Based on this, in this Python interpreter integration device, an innovation is made to this problem, which allows multiple programming languages ​​to integrate the Python interpreter. The specific implementation process is as follows: First, the basic code of this device is developed in C++. In this C++ developed code, the core provides a function named ExecuteCode(PythonCode), which accepts Python code, executes Python code, and returns after obtaining the execution result.

[0105] Secondly, through the glue language SWIG, the above C++ code is converted into codes of programming languages ​​such as Java interface code, Go interface code, and corresponding dynamic library files.

[0106] Finally, other programming languages ​​(such as Java) only need to load the above dynamic library files and call the above Java interface to complete the language's parsing of the Python interpreter.

[0107] Based on the above content, it can be seen that the code processing method in one or more embodiments of this specification provides a high-performance, easy-to-use, and multi-programming language integrated device for Python interpreters; the device proposes a new multi-process mode, and each process has an independent Python interpreter built in. This design allows Python interpreters in different processes to run independently of each other, completely free from the constraints of GIL, thereby achieving true parallel execution of Python code.

[0108] The advantage of this architecture is that it is not only applicable to Java, but can also be widely used in the integration of many other programming languages ​​such as C++, Go, Ruby, etc. By providing an independently running Python interpreter process for each integrated language, the present invention significantly improves the execution efficiency, especially in application scenarios that need to process large amounts of data or perform complex computing tasks. In addition, the multi-process architecture enhances the robustness of the system, and the failure of a single process will not affect other processes, thereby improving the overall system stability and reliability.

[0109] In summary, this innovation not only solves the performance bottleneck problem of Python in a multi-threaded environment, but also provides an efficient and scalable solution for the integration of multiple programming languages ​​with Python. This allows the powerful functions of Python to be better integrated with applications in other languages, expanding the application potential of Python in multiple fields.

[0110] Compared with the above solutions, the code processing method in this manual can break through the original Python GIL limitation of only a single thread access through the subprocess pool, greatly improving performance. In addition, the code processing method in this manual supports use in multiple programming languages, greatly expanding its popularity.

[0111] Compared with all the current Java solutions for executing Python code on the market (limited by GIL), the code processing method in this manual uses a multi-process mode to build an independent Python interpreter for each process, greatly improving the efficiency of Java executing Python code. In other words, the current solutions on the market are basically limited by Python's GIL, and the performance is greatly affected. This solution has made innovations in this regard through the subprocess pool solution, and the performance can be greatly improved.

[0112] In addition, the two major methods of execute and evaluate in the Python interpreter API are combined and automatically selected based on the statement, so users do not need to worry about which method their code calls.

[0113] Furthermore, the code processing method in this manual supports the built-in Python interpreter for multiple programming languages; that is to say, the current solutions on the market are basically aimed at a specific programming language, and this solution supports multiple programming languages ​​at the same time and has high adaptability.

[0114] In addition, this solution can also support Python 3.x version, compared to jython which only supports 2.x version, which overall matches the long-term plan of the Python language.

[0115] Corresponding to the above method embodiment, this specification also provides a code processing device embodiment, Figure 8 FIG. 1 shows a schematic diagram of the structure of a code processing device provided by an embodiment of the present specification. Figure 8 As shown, the device comprises: A first unit determination module 802 is configured to determine a target code and a target code processing unit corresponding to the target code; A second unit determination module 804 is configured to use the target code processing unit to determine a target code processing subunit that processes the target code from a plurality of code processing subunits, wherein the code processing subunit is generated according to the target code processing unit and each code processing subunit has a corresponding code interpreter; A code sending module 806 is configured to send the target code to the target code processing subunit using the target code processing unit; The code processing module 808 is configured to process the target code using the target code processing subunit and the target code interpreter corresponding to the target code processing subunit to obtain a code processing result corresponding to the target code.

[0116] Optionally, the code processing module 808 is further configured to: Utilizing the target code processing subunit, receiving the target code, and calling the target code interpreter, parsing the target code according to the abstract syntax tree, and determining a plurality of code statements in the target code; A target code statement containing an expression is determined from the multiple code statements, and the target code statement is executed to obtain a code processing result corresponding to the target code.

[0117] Optionally, the code processing module 808 is further configured to: A to-be-executed code statement that does not contain an expression is determined from the multiple code statements, and the to-be-executed code statement is executed.

[0118] Optionally, the code processing device further includes a result sending module, which is configured to: Sending the code processing result to the target code processing unit; The target code processing unit is used to send the code processing result to a result acquisition unit.

[0119] Optionally, the code processing device further includes a unit exit module configured to: When receiving an exit instruction for the target code processing unit, exiting the plurality of code processing sub-units by sending the exit instruction to the plurality of code processing sub-units using the target code processing unit; When it is determined that the exit of the plurality of code processing sub-units is completed, the target code processing unit is exited.

[0120] Optionally, the first unit determination module 802 is further configured to: Receiving a target code sent by a code providing unit, wherein the target code is sent by the code providing unit by calling a code providing component, and the code providing component includes a function and an interface; Based on the target code, dynamic library files corresponding to the target code processing unit and the multiple code processing sub-units are loaded, and the target code processing unit and the multiple code processing sub-units are controlled to process the target code.

[0121] Optionally, the target code processing unit is a main process, the code processing sub-unit is a sub-process, the sub-process includes multiple threads, and the sub-process is generated by replicating the main process.

[0122] The code processing device provided by one or more embodiments of the present specification can generate multiple code processing sub-units according to the target code processing unit, and each code processing sub-unit maintains a corresponding code interpreter; therefore, in the process of executing the code, the target code is sent to the target code processing sub-unit that processes the target code by using the target code processing unit, and the target code processing sub-unit and the target code interpreter maintained by it are used to execute the target code in parallel to obtain the code processing result corresponding to the target code; thereby, by maintaining a code interpreter for each code processing sub-unit, the limitation of the global interpreter lock is broken, so that multiple code processing sub-units can execute code in parallel at the same time, thereby improving the efficiency of code execution.

[0123] The above is a schematic scheme of a code processing device of this embodiment. It should be noted that the technical scheme of the code processing device and the technical scheme of the above code processing method belong to the same concept, and the details not described in detail in the technical scheme of the code processing device can be referred to the description of the technical scheme of the above code processing method.

[0124] Fig. 9 The structure block diagram of a computing device 900 provided according to an embodiment of the present specification is shown. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0125] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0126] In one embodiment of the present specification, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0127] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 900 may also be a mobile or stationary server.

[0128] The processor 920 is used to execute the following computer program / instruction, which implements the steps of the above-mentioned code processing method when executed by the processor.

[0129] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the code processing method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the code processing method embodiment.

[0130] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned code processing method when executed by a processor.

[0131] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the code processing method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the code processing method embodiment.

[0132] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned code processing method when executed by a processor.

[0133] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned code processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned code processing method.

[0134] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0135] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0136] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0137] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0138] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A code processing method, characterized in that: include: Determine the target code and the target code processing unit corresponding to the target code; Determining, by using the target code processing unit, a target code processing subunit for processing the target code from a plurality of code processing subunits, wherein the code processing subunit is generated according to the target code processing unit and each code processing subunit has a corresponding code interpreter; Using the object code processing unit, sending the object code to the object code processing subunit; The target code is processed by using the target code processing subunit and the target code interpreter corresponding to the target code processing subunit to obtain a code processing result corresponding to the target code.

2. The code processing method according to claim 1, characterized in that: The target code processing subunit and the target code interpreter corresponding to the target code processing subunit are used to process the target code to obtain a code processing result corresponding to the target code, including: Utilizing the target code processing subunit, receiving the target code, and calling the target code interpreter, parsing the target code according to the abstract syntax tree, and determining a plurality of code statements in the target code; A target code statement containing an expression is determined from the multiple code statements, and the target code statement is executed to obtain a code processing result corresponding to the target code.

3. The code processing method according to claim 2, characterized in that: The calling of the target code interpreter, parsing the target code according to the abstract syntax tree, and determining a plurality of code statements in the target code further comprises: A to-be-executed code statement that does not contain an expression is determined from the multiple code statements, and the to-be-executed code statement is executed.

4. The code processing method according to claim 1, characterized in that: After the target code is processed by the target code processing subunit and the target code interpreter corresponding to the target code processing subunit to obtain the code processing result corresponding to the target code, the method further includes: Sending the code processing result to the target code processing unit; The target code processing unit is used to send the code processing result to a result acquisition unit.

5. The code processing method according to claim 1, characterized in that: The method further comprises: When receiving an exit instruction for the target code processing unit, exiting the plurality of code processing sub-units by sending the exit instruction to the plurality of code processing sub-units using the target code processing unit; When it is determined that the exit of the plurality of code processing sub-units is completed, the target code processing unit is exited.

6. The code processing method according to claim 1, characterized in that: The determining of the target code comprises: Receiving a target code sent by a code providing unit, wherein the target code is sent by the code providing unit by calling a code providing component, and the code providing component includes a function and an interface; Based on the target code, dynamic library files corresponding to the target code processing unit and the multiple code processing sub-units are loaded, and the target code processing unit and the multiple code processing sub-units are controlled to process the target code.

7. The code processing method according to any one of claims 1 to 6, characterized in that: The target code processing unit is a main process, the code processing sub-unit is a sub-process, the sub-process includes multiple threads, and the sub-process is generated by replicating the main process.

8. A computing device, characterized in that include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the code processing method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: It stores a computer program / instruction, which, when executed by a processor, implements the steps of the code processing method described in any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program / instruction, which, when executed by a processor, implements the steps of the code processing method according to any one of claims 1 to 7.

Images