Compiling method and device for real-time computing task codes

Abstract

The compiling method for the real-time computing task code is applied to a real-time computing platform, and a process used for executing a compiling task corresponding to the real-time computing task code submitted by a user runs on the real-time computing platform. Wherein the process is a process based on a class isolation framework; comprising the steps of obtaining a compiling request corresponding to a real-time calculation task code submitted by a user; analyzing the compilation request, determining at least one dependency item on which a compilation task corresponding to the real-time calculation task code depends, and dynamically creating a thread for executing the compilation task in the process; and running the thread, loading the class contained in the at least one dependency item in an isolated manner based on a class loading isolation mechanism supported by the class isolation framework, and executing the loaded class so as to complete compiling for the real-time computing task code.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of real-time computing, and particularly relate to a method and device for compiling real-time computing task code. Background Art

[0002] In some real-time computing platforms, in addition to providing real-time computing services open to users, it is usually also allowed that when users utilize the computing power of the real-time computing platform to execute some real-time computing tasks, they submit some custom real-time computing codes related to the real-time computing tasks.

[0003] When the server of the real-time computing platform processes the real-time computing requests submitted by users, it usually also needs to compile the real-time computing codes submitted by users, and convert the real-time computing codes submitted by users into a form that can be recognized by the computing engine of the real-time computing platform; for example, taking the Flink platform as an example, it usually compiles the real-time computing codes submitted by users into the form of an execution plan that can be recognized by the computing engine of the Flink platform.

[0004] However, in practical applications, the logic for compiling the real-time computing codes submitted by users usually needs to rely on the compilation function in the computing engine of the real-time computing platform. In order to ensure the correctness, isolation, and security of different compilation requests, it usually consumes a large amount of computing resources on the real-time computing platform, which may result in limited single-machine processing capabilities of the server of the real-time computing platform, poor system stability, and greatly affect the overall throughput. Summary of the Invention

[0005] This specification proposes a method for compiling real-time computing task code, which is applied to a real-time computing platform on which a process for executing a compilation task corresponding to the real-time computing task code submitted by a user is running; wherein, the process is a process based on a class isolation framework; the method includes:

[0006] Obtain a compilation request corresponding to the real-time computing task code submitted by the user;

[0007] Parse the compilation request, determine at least one dependency on which the compilation task corresponding to the real-time computing task code depends, and dynamically create a thread for executing the compilation task within the process;

[0008] Run the thread, isolate and load the classes included in the at least one dependency based on the class loading isolation mechanism supported by the class isolation framework, and execute the loaded classes to complete the compilation of the real-time computing task code.

[0009] Optionally, the class loading isolation mechanism supported by the class isolation framework includes creating different class loaders for each dependency on which the code depends, and loading the classes included in each dependency on which the code depends through different class loaders, so as to implement the class isolation mechanism for the classes included in each dependency;

[0010] Running the thread, and based on the class loading isolation mechanism supported by the class isolation framework, isolating and loading the classes included in the at least one dependency, including:

[0011] Creating different class loaders for each dependency in the at least one dependency;

[0012] Running the thread, and based on the class loaders created for each dependency respectively, loading the classes included in each dependency respectively.

[0013] Optionally, the process has created different class loaders for the library files related to the various code compilation functions provided by the real-time computing platform in advance; the class loaders created for the library files related to the various code compilation functions are cached in the memory of the process; the at least one dependency includes a custom dependency provided by the user; and at least one library file related to at least one code compilation function provided by the real-time computing platform;

[0014] Running the thread, and based on the class loaders created for each dependency respectively, loading the classes included in each dependency respectively, including:

[0015] Dynamically loading the class loaders respectively corresponding to the various library files in the at least one library file from the class loaders cached in the memory of the process, and dynamically creating a class loader for the custom dependency;

[0016] Running the thread, and based on the dynamically created class loader, loading the classes included in the custom dependency, and based on the dynamically loaded class loaders, loading the classes included in the various library files respectively.

[0017] Optionally, the library file related to any one code compilation function provided by the real-time computing platform includes multiple versions of library files corresponding to multiple different function versions of this code compilation function; the class loader cached in the memory of the process is a common class loader shared by the multiple versions of library files corresponding to the same code compilation function; wherein, the common class loader is not configured with a class loading path;

[0018] Dynamically loading the class loaders respectively corresponding to the various library files in the at least one library file from the class loaders cached in the memory of the process, including:

[0019] Determine the versions of the respective library files;

[0020] From the class loaders cached in the memory of the process, dynamically load general class loaders respectively corresponding to the versions of the respective library files in the at least one library file, and dynamically configure the class loading paths of the respective library files into the corresponding general class loaders.

[0021] Optionally, the at least one library file includes at least one basic library file and at least one extension library file related to the code compilation function provided by the real-time computing platform;

[0022] The custom dependencies include:

[0023] The source code corresponding to the real-time computing task code; and, the functions customized by the user on which the source code depends.

[0024] Optionally, the real-time computing platform includes the Flink platform; the basic library file includes the full library files stored in the lib directory of the Flink platform; the extension library file includes the partial library files specified by the user and stored in the opt directory of the Flink platform.

[0025] Optionally, the process includes a Jvm process; the class isolation framework includes the koupleless framework; plugins built based on class loaders respectively created for the library files related to the respective code compilation functions are cached in the memory of the process;

[0026] From the class loaders cached in the memory of the process, dynamically load class loaders respectively corresponding to the respective library files in the at least one library file, and dynamically create a class loader for the custom dependencies, including:

[0027] Dynamically load at least one plugin built based on class loaders respectively corresponding to the respective library files in the at least one library file from the plugins cached in the memory of the process; in response to the completion of the loading of the at least one plugin, further dynamically create a class loader for the custom dependencies, and further build a Serverless application module running based on the thread based on the dynamically created class loader; wherein, the Serverless application module is used to execute the compilation task;

[0028] Run the thread, based on the dynamically created class loader, load the classes included in the custom dependencies, and based on the dynamically loaded class loaders, respectively load the classes included in the respective library files, including:

[0029] Run the Serverless application module based on the thread, so as to load the classes included in the custom dependencies based on the dynamically created class loader, and call the at least one dynamically loaded plugin during the running of the Serverless application module, so as to load the classes included in the respective library files based on the class loaders corresponding to the respective library files.

[0030] Optionally, the plugin includes a plugin created based on the Ark component in the koupleless framework; the Serverless application module includes a Biz module created based on the Ark component in the koupleless framework.

[0031] Optionally, the method further includes:

[0032] Periodically scan the idle threads running within the process;

[0033] Determine whether the scanned idle thread is a thread used to execute the compilation task;

[0034] If the scanned idle thread is a thread used to execute the compilation task, further determine whether the compilation request corresponding to the compilation task executed by the thread is an invalid compilation request; if so, recycle the thread.

[0035] Optionally, the memory in the process for caching the class loaders respectively created for the at least one library file is the heap memory of the Jvm process; metadata generated during the process of the threads running in the process loading the classes included in the at least one dependency is also cached in the off-heap memory of the process;

[0036] The method further includes:

[0037] Periodically detect whether the storage space occupied by the metadata cached in the off-heap memory reaches a threshold;

[0038] If so, restart the Jvm process.

[0039] This specification also proposes a compilation device for real-time computing task code, which is applied to a real-time computing platform, and a process for executing a compilation task corresponding to the real-time computing task code submitted by a user is running on the real-time computing platform; wherein, the process is a process based on a class isolation framework; the device includes:

[0040] An acquisition module, which acquires a compilation request corresponding to the real-time computing task code submitted by a user;

[0041] A parsing module that parses the compilation request, determines at least one dependency on which the compilation task corresponding to the real-time computing task code depends, and dynamically creates a thread for executing the compilation task within the process;

[0042] A loading module that runs the thread, isolates and loads the classes included in the at least one dependency based on the class loading isolation mechanism supported by the class isolation framework, and executes the loaded classes to complete the compilation of the real-time computing task code.

[0043] In the above embodiment, when the real-time computing platform executes the compilation task corresponding to the real-time computing task code submitted by the user, on the one hand, by dynamically creating a thread for executing the compilation task within the process based on the class isolation framework and executing the compilation task based on the thread inside the process, the lighter weight characteristic of the thread can be fully utilized, the resource consumption during the execution of the compilation task can be reduced, and the execution speed of the compilation task can be improved; on the other hand, during the process of running the thread to load the dependencies on which the compilation task depends, by using the class loading isolation mechanism supported by the class isolation framework of the process to isolate and load the classes included in the dependencies, not only can the correct loading of the dependencies be ensured, but also the isolation and security between different compilation tasks can be taken into account. Description of the Drawings

[0044] To more clearly illustrate the technical solutions of the embodiments of this specification, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments recorded in this specification. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0045] Figure 1 is a flowchart of a method for compiling real-time computing task code shown in an embodiment of this specification;

[0046] Figure 2 is a schematic diagram of a loading framework of a class loader shown in an embodiment of this specification;

[0047] Figure 3 is a schematic diagram of a three-layer task execution framework based on Koupleless designed for the Jvm process shown in an embodiment of this specification;

[0048] Figure 4 is a flowchart of a method for executing a compilation task shown in an embodiment of this specification;

[0049] Figure 5 is a schematic structural diagram of an electronic device shown in an embodiment of this specification;

[0050] Figure 6 It is a block diagram of a compilation device for real-time computing task code shown in an embodiment of this specification. Detailed implementation manners

[0051] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this specification.

[0052] In the related art, when a real-time computing platform compiles real-time computing code submitted by a user, in order to ensure the correctness, isolation, and security of different compilation requests, the commonly used process model in the industry is usually adopted to process the compilation requests. Based on the common process model, every time the server of the real-time computing platform obtains a compilation request, it will create an independent process for this compilation request and execute the compilation task corresponding to this compilation request through this process.

[0053] However, using the process model to process compilation requests, although it can utilize the isolation feature of the process itself to ensure the correctness, isolation, and security of different compilation requests, creating an independent process for each compilation request will waste a large amount of computing resources on the real-time computing platform.

[0054] For example, taking the Flink real-time computing platform as an example, the Flink platform will create an independent Jvm (Java Virtual Machine) process for each compilation request and execute the compilation task corresponding to this compilation request based on this Jvm process. However, starting a Jvm process usually requires executing a series of complex processes, which will result in a long time consumption even for a compilation request that only involves some lightweight compilation operations, seriously affecting the experience of users submitting real-time computing task code to the Flink platform.

[0055] Moreover, since starting a Jvm process usually consumes a large amount of real-time computing resources (such as memory resources and CPU resources) on the Flink platform, it may lead to limited single-machine processing capacity of the server of the real-time computing platform and poor system stability, thus greatly affecting the overall throughput of the platform.

[0056] Based on this, this specification proposes a technical solution that, on a real-time computing platform, executes compilation tasks based on a more lightweight thread model and, based on the class isolation framework of the process, isolates and loads the classes included in the dependencies on which the compilation tasks depend.

[0057] Please refer to Figure 1 , Figure 1 which is a flowchart of a method for compiling real-time computing task code shown in this specification, applied to a real-time computing platform on which a process for executing a compilation task corresponding to the real-time computing task code submitted by the user is running; wherein, the process is a process based on a class isolation framework; the method includes the following execution process:

[0058] Step 102, obtain a compilation request corresponding to the real-time computing task code submitted by the user;

[0059] The above real-time computing platform may specifically include any form of computing platform for executing real-time computing tasks. The programming language and system framework adopted by the above real-time computing platform are not particularly limited in this specification.

[0060] For example, in an illustrated embodiment, the above real-time computing platform may specifically include a Flink platform; wherein, the Flink platform is a computing platform for executing real-time computing tasks built based on the open-source Apache Flink framework.

[0061] The above compilation request may specifically include any form of request for triggering the compilation of the real-time computing task code submitted by the user.

[0062] For example, in an example, the compilation request may specifically be a code submission request submitted by the user, and the code submission request may specifically include the real-time computing task code that the user needs to submit.

[0063] In another example, the compilation request may specifically also be a compilation request triggered and generated by the real-time computing platform for requesting the compilation of the real-time computing task code after the real-time computing platform obtains the real-time computing task code submitted by the user.

[0064] In practical applications, in addition to providing real-time computing services to users, the above real-time computing platform generally also allows users to submit some user-defined real-time computing task codes related to real-time computing tasks when using the computing power of the real-time computing platform to execute some real-time computing tasks.

[0065] After obtaining the real-time computing task code related to the real-time computing task submitted by the user, the real-time computing platform can parse the real-time computing task code submitted by the user to determine whether the real-time computing task code submitted by the user is user-defined code, or whether the real-time computing task code contains user-defined code.

[0066] If the real-time computing task code submitted by the user is user-defined code, or the real-time computing task code contains user-defined code, at this time, the real-time computing task code cannot be directly executed on the real-time computing platform, but needs to use the code compilation function provided in the computing engine of the real-time computing platform to compile the real-time computing task code into a form that can be recognized by the computing engine of the real-time computing platform.

[0067] Among them, the real-time computing platform compiles the real-time computing task code submitted by the user, which usually means converting the code generated based on a high-level programming language submitted by the user into a low-level form that can be recognized and efficiently executed by the real-time computing platform; among them, the compiled code can be a low-level operation sequence or a low-level operation graph. For different real-time computing platforms, the specific form of the compiled code usually also has certain differences.

[0068] For example, taking the above real-time computing platform as the Flink platform, the Flink platform usually compiles the code generated based on a high-level programming language (such as the Java language) submitted by the user into the form of an "execution plan" that can be recognized by the computing engine of the Flink platform.

[0069] Another example, taking the above real-time computing platform as the Storm platform, the Storm platform usually compiles the code generated based on a high-level programming language submitted by the user into the form of a "topology" that can be recognized by the computing engine of the Storm platform.

[0070] In an illustrated embodiment, the above compilation request may specifically be a code submission request submitted by the user; among them, the code submission request may specifically include the real-time computing task code that the user needs to submit.

[0071] In this case, after the server of the real-time computing platform obtains the code submission request submitted by the user, it can parse the real-time computing task code included in the code submission request to determine whether the real-time computing task code submitted by the user is user-defined code, or whether the real-time computing task code contains user-defined code. If the real-time computing task code submitted by the user is user-defined code, or the real-time computing task code contains user-defined code, at this time, the real-time computing platform can use the code compilation function provided in the computing engine to compile the real-time computing task code.

[0072] In another illustrated embodiment, the compilation request may specifically be a compilation request triggered and generated by the real-time computing platform after obtaining the real-time computing task code submitted by the user, for requesting compilation of the real-time computing task code; wherein, the real-time computing task code that the user needs to submit may also be specifically included in the compilation request.

[0073] In this case, after the server of the real-time computing platform obtains the code submission request submitted by the user, it can parse the real-time computing task code included in the code submission request to determine whether the real-time computing task code submitted by the user is user-defined code, or whether the real-time computing task code contains user-defined code. If the real-time computing task code submitted by the user is user-defined code, or the real-time computing task code contains user-defined code, at this time, the real-time computing platform can trigger and generate a compilation request for requesting compilation of the real-time computing task code, obtain the generated compilation request, and process the compilation request; for example, the triggered and generated compilation request can be added to a queue, and then the compilation request is read from the queue for processing. Then, the code compilation function provided in the computing engine can be used to compile the real-time computing task code.

[0074] Step 104, parse the compilation request, determine at least one dependency on which the compilation task corresponding to the real-time computing task code depends, and dynamically create a thread for executing the compilation task within the process;

[0075] After the real-time computing platform obtains the compilation request corresponding to the real-time computing task code submitted by the user, it can also parse the compilation request to determine at least one dependency on which the compilation task corresponding to the real-time computing task code included in the compilation request depends.

[0076] It should be noted that the above dependencies refer to various external resources required for the normal operation of the above compilation task.

[0077] For example, in practical applications, the above various external resources may specifically include external library files, configuration files, data sources (such as databases), network services, environment variables, and so on.

[0078] In an illustrated embodiment, at least one dependency on which a compilation task corresponding to the real-time computing task code included in the compilation request depends may specifically include a custom dependency provided by a user; and at least one library file related to at least one code compilation function provided by the real-time computing platform.

[0079] Among them, the above custom dependency may specifically include any form of external resource customized by the user required for the normal operation of the above compilation task.

[0080] In an illustrated embodiment, the custom dependency provided by the user may specifically include the source code corresponding to the real-time computing task code submitted by the user; and the user-defined UDF (User Define Function) on which the source code depends.

[0081] The library file related to the code compilation function provided by the above real-time computing platform may also specifically include the library file related to any type of code compilation function that the above real-time computing platform can provide.

[0082] In an illustrated embodiment, the code compilation functions that the above real-time computing platform can provide may generally include a basic compilation function and an extended compilation function introduced on the basis of the basic compilation function. Correspondingly, the library files related to the code compilation functions that the above real-time computing platform can provide may generally include a basic library file corresponding to the above basic compilation function and an extended library file corresponding to the above extended compilation function. It should be noted that the above extended library file may specifically be a library file used to expand the capabilities of the above basic compilation function without modifying the source code of the above basic compilation function.

[0083] In this case, at least one library file included in at least one dependency on which the above compilation task depends may specifically include at least one basic library file and at least one extended library file related to at least one code compilation function provided by the real-time computing platform.

[0084] For example, taking the above real-time computing platform as the Flink platform, in the framework of the Flink platform, there are usually two directories: Flink-lib and Flink-opt. Among them, the Flink-lib directory is used to store the JAR files necessary for the operation of some core functions (such as code compilation functions) of the Flink platform. The Flink-opt directory is used to store optional JAR files. These optional JAR files are usually not the JAR files necessary for the operation of some core functions of the Flink platform, but JAR files that can expand the capabilities of these core functions without modifying the code of these core functions, so as to enhance the core functions.

[0085] In this case, the above at least one basic library file may specifically include all the library files stored in the lib directory of the Flink platform; and the above at least one extended library file may specifically include some of the library files specified by the user stored in the opt directory of the Flink platform. For example, the JAR files related to the code compilation function stored in the Flink-opt directory of the Flink platform usually include JAR files corresponding to common connector, backend and other plugin packages. Then, in the application scenario of code compilation, the above at least one extended library file may specifically include the JAR files corresponding to the connector, backend and other plugin packages stored in the opt directory of the Flink platform.

[0086] Of course, since the compilation requirements of the compilation tasks corresponding to different compilation requests usually vary, and the code compilation functions provided on the real-time computing platform they depend on usually also vary; therefore, for the compilation tasks corresponding to different compilation requests, the library files they depend on may also be different to some extent.

[0087] It should be noted that since creating an independent process for each compilation request to execute the compilation task corresponding to the compilation request usually wastes a large amount of computing resources on the real-time computing platform; for example, a compilation request may only involve some lightweight compilation operations. For such a compilation request, creating an independent process to execute the compilation task corresponding to the compilation request will obviously unreasonably occupy a large amount of computing resources on the real-time computing platform, thus causing waste of computing resources; therefore, in this specification, the process model can no longer be used to process compilation requests, but the thread model is adopted, and different threads are dynamically created for different compilation requests within a process to process the compilation requests.

[0088] The server of the above real-time computing platform can still run a process on the above real-time computing platform for executing a compilation task corresponding to the real-time computing task code submitted by the user.

[0089] Among them, the process can specifically be a process based on a class isolation framework. In this process, a class isolation framework can be adopted to ensure the isolation of different classes to be loaded during the running of this process.

[0090] For example, taking the above real-time computing platform as the Flink platform, the above process can specifically include a JVM process, and the above class isolation framework can specifically include the koupleless framework. It should be noted that the concept of class mentioned in this specification is the core concept in object-oriented programming and can be used to define a set of attributes and methods.

[0091] When the real-time computing platform obtains a compilation request, it can dynamically create a thread for executing the compilation task corresponding to this compilation request in this process; among them, this thread is specifically used to execute the compilation task corresponding to the compilation request. For different compilation requests, different threads can be created respectively.

[0092] For example, in practical applications, the thread dynamically created for the obtained compilation request in this process can specifically be an asynchronous thread. An asynchronous thread refers to a thread that can run independently of the main thread or other threads during the program execution and does not block the execution of other threads.

[0093] It should be supplemented and explained that the execution order of the steps of parsing the compilation request after the real-time computing platform obtains the compilation request and the step of dynamically creating a thread for this compilation request in the above process will not be specifically limited in this specification;

[0094] For example, in practical applications, the step of parsing the compilation request can be executed in parallel with the step of dynamically creating a thread for this compilation request in the above process, or can be executed in a certain order; for example, the step of parsing the compilation request can be executed first, and then the step of dynamically creating a thread for this compilation request in the above process; or, the step of dynamically creating a thread for this compilation request in the above process can be executed first, and then the step of parsing the compilation request.

[0095] Step 106, run the thread, isolate and load the classes included in the at least one dependency based on the class loading isolation mechanism supported by the class isolation framework, and execute the loaded classes to complete the compilation of the real-time computing task code;

[0096] When, based on the compilation request obtained through parsing, it is determined that there are at least one dependency on which the compilation task corresponding to the real-time computing task code included in the compilation request depends, and after a thread for executing this compilation task is dynamically created within the above process, this thread can be run to execute the compilation task corresponding to this compilation request. During the execution of this compilation task, the classes included in the above at least one dependency can also be loaded in isolation based on the class loading isolation mechanism supported by the class isolation framework adopted by the above process.

[0097] Among them, the above class loading isolation mechanism specifically refers to the specific class isolation scheme adopted by the class isolation framework adopted by the above process to achieve isolated loading of classes. In this specification, no special limitation will be imposed on the class loading isolation mechanism adopted by the class isolation framework. In practical applications, the class loading isolation mechanisms supported by different class isolation frameworks can be the same or different.

[0098] In an illustrated implementation manner, the class loading isolation mechanism supported by the above class isolation framework may specifically include creating different class loaders for each dependency on which the code depends, and loading the classes included in each dependency on which the code depends through different class loaders, so as to implement the class isolation mechanism for the classes included in the above each dependency.

[0099] In this case, during the process of running the above thread to execute the compilation task corresponding to this compilation request, different class loaders can be created respectively for each dependency among the at least one dependency on which this compilation task depends, and then this thread can be run. Based on the class loaders created respectively for the above each dependency, the classes included in the above each dependency are loaded respectively, so as to achieve class isolation among the classes included in the above each dependency.

[0100] For example, in an illustrated implementation manner, the at least one dependency on which the compilation task corresponding to the real-time computing task code included in this compilation request depends may specifically include a custom dependency provided by the user; and at least one library file related to at least one code compilation function provided by the real-time computing platform.

[0101] In this case, the above process can create different class loaders for each of the above custom dependency and each library file among the above at least one library file. For example, in practical applications, a class loader can be created separately for the above custom dependency, and a class loader can be created respectively for each library file among the above at least one library file.

[0102] In the process of running the above thread to execute the compilation task corresponding to the compilation request, different class loaders can be used to load the classes included in the above custom dependencies and each library file in the above at least one library file respectively.

[0103] It should be noted that when the class loader loads the classes included in each dependency, specifically, it can load the classes included in each dependency based on the class loading path (classpath) configured in the class loader. The class loading path configured in the class loader can be statically configured in the class loader at the stage of creating the class loader, or can be dynamically loaded into the class loader when the class loader is called, which is not specifically limited in this specification.

[0104] In practical applications, since the library files related to the compilation functions provided by the real-time computing platform that the compilation tasks corresponding to different compilation requests depend on are usually some general library files that need to be frequently used by different compilation requests, in order to improve the usage efficiency of these general library files, the class loaders created for these general library files can be cached in advance in the memory of the above process in a way of staying resident in memory.

[0105] Please refer to Figure 2 , Figure 2 which is a schematic diagram of a loading framework of a class loader shown in this specification.

[0106] In Figure 2 the framework shown, the above process can create different class loaders for the library files related to each code compilation function provided by the real-time computing platform respectively, and cache the created class loaders in the memory of the process in a way of staying resident in memory.

[0107] For example, taking the above process as a Jvm process, the Jvm process can create different class loaders for the library files related to each code compilation function provided by the real-time computing platform at the stage of process initialization, and then cache the created class loaders in the heap memory of the Jvm process. Among them, the heap memory is the largest part of the Jvm memory and is shared by all threads.

[0108] In this case, in the process of running the above thread to execute the compilation task corresponding to the compilation request, on the one hand, the class loaders respectively corresponding to each library file in the above at least one library file that the compilation task depends on can be dynamically loaded from the class loaders cached in the memory of the above process;

[0109] For example, taking the above process as the Jvm process, after dynamically creating a thread for executing the compilation task corresponding to the compilation request within the above process, an independent stack memory is usually allocated for this thread as the private stack space of the thread. This stack memory is specifically used to store basic type variables required for the thread to run (such as int, float, etc.), references to objects, local variables for method calls, and so on. When executing a task based on this thread, all local variables and parameters required for executing the task are usually stored in the stack memory of this thread. In this case, dynamically loading class loaders corresponding to each of the at least one library file on which the compilation task depends from the class loaders cached in the memory of the above process usually means loading the class loaders corresponding to each of the at least one library file on which the compilation task depends cached in the heap memory of the Jvm process into the stack memory of this thread respectively. After the loading is completed, when running this thread subsequently, this thread can load the classes contained in each of the above library files through the class loaders loaded in the stack memory.

[0110] Of course, during the process of dynamically loading class loaders corresponding to each of the at least one library file on which the compilation task depends from the class loaders cached in the memory of the above process, if some of the at least one library file are not cached in the memory of the above process, a loading failure will occur. At this time, the above process can dynamically create class loaders for the missing part of the library files.

[0111] On the other hand, the above process can also dynamically create a request-level class loader for the user-provided custom dependencies on which the compilation task corresponding to the compilation request depends.

[0112] For example, still taking the above process as the Jvm process, when dynamically creating a class loader for the above custom dependencies on which the compilation task depends, specifically, a class loader for the above custom dependencies on which the compilation task depends can be dynamically created in the heap memory of the Jvm process first, and then the dynamically created class loader can be loaded from the heap memory into the stack memory of the above thread.

[0113] Please continue to refer to Figure 2 , in practical applications, the user-provided custom dependencies on which the above compilation task depends can specifically include the source code corresponding to the real-time computing task code submitted by the user; and, the user-defined UDF on which the source code depends. The at least one library file included in the at least one dependency on which the above compilation task depends can specifically include at least one basic library file and at least one extension library file related to at least one code compilation function provided by the real-time computing platform.

[0114] In this case, the class loaders cached in the above process may specifically include the class loader corresponding to the above source code and the class loader corresponding to the above user-defined UDF; and, the class loaders corresponding to the above at least one basic library file and the class loaders corresponding to the above at least one extended library file respectively.

[0115] Among them, before the above process dynamically creates a class loader for the above custom dependencies, it may first confirm whether the class loaders corresponding to each of the above at least one library file on which the compilation task depends have been loaded from the memory of the above process; if all are loaded, it indicates that the library files on which the compilation task depends are ready, and at this time, the above process can start to dynamically create a class loader for the above custom dependencies.

[0116] In this way, it can be ensured that after the library files on which the compilation task depends are all ready, a class loader is created for the custom dependencies provided by the user, thereby avoiding the problem of compilation failure caused by the unprepared library files on which the compilation task depends.

[0117] After dynamically loading the class loaders corresponding to each of the above at least one library file on which the compilation task depends from the memory of the above process and dynamically creating a class loader for the above custom dependencies on which the compilation task depends, at this time, the above thread can be run to load the classes included in the above custom dependencies based on the class loader dynamically created for the above custom dependencies, and to load the classes included in the above respective library files based on the class loaders dynamically loaded from the memory of the above process.

[0118] Please continue to refer to Figure 2 , taking the above real-time computing platform as the Flink platform as an example, the above at least one basic library file may specifically include all the library files stored in the lib directory of the Flink platform; and the above at least one extended library file may specifically include some library files specified by the user stored in the opt directory of the Flink platform.

[0119] For example, as Figure 2 shown, the above at least one extended library file may specifically include the library files corresponding to plugin packages such as connector and backend stored in the opt directory of the Flink platform. At this time, the class loaders cached in the above process corresponding to the above at least one basic library file may specifically include the class loader corresponding to all the library files stored in the lib directory, and the class loaders corresponding to the plugin packages such as connector and backend stored in the above opt directory respectively.

[0120] In one of the illustrated embodiments, please continue to refer to Figure 2 , in practical applications, any library file related to the code compilation function provided by the real-time computing platform may specifically include multiple versions of library files corresponding to multiple different functional versions of this code compilation function. In this case, the above process may create different class loaders for these multiple versions of library files respectively, and in the memory of this process, cache and manage these class loaders according to different functional versions.

[0121] For example, as Figure 2 shown, still taking the above real-time computing platform as the Flink platform as an example, assume that the code compilation functions provided by the Flink platform have a total of three versions, namely version 1, version 2, and version 3. At this time, each version can have an independent lib directory and opt directory; in this case, the above process may create different class loaders for the full set of library files stored in the lib directory of each version and the plugin packages such as connectors and backends stored in the opt directory, and then cache and manage them in the memory of the process according to different functional versions.

[0122] In one of the illustrated embodiments, when any library file related to the code compilation function provided by the real-time computing platform may specifically include multiple versions of library files corresponding to multiple different functional versions of this code compilation function, the above process may also not need to distinguish versions and create different class loaders for these multiple versions of library files respectively, but create a shared general class loader for the multiple versions of library files corresponding to the same code compilation function. Among them, the class loading path may not be configured by default in this general class loader.

[0123] In this case, each class loader cached in the memory of the above process may specifically be a general class loader shared by multiple versions of library files corresponding to the same code compilation function. When dynamically loading the class loaders respectively corresponding to each library file in the at least one library file from the class loaders cached in the memory of the above process, it may specifically first determine the versions of the above library files to be loaded, and dynamically load the general class loaders respectively corresponding to the versions of the above library files from the class loaders cached in the memory of the above process, and then dynamically configure the class loading paths of the above library files into the corresponding general class loaders respectively. That is to say, for the general class loader, the class loading path may not be configured by default, but the required class loading path may be loaded in real time during the usage stage after the general class loader is loaded.

[0124] Of course, after the compilation task is completed, the class loading path dynamically configured in the general class loader can also be cleared, and the general class loader can be restored to the default state without a configured class loading path.

[0125] In this way, only one general class loader shared by multiple versions of the library file needs to be created for each library file, rather than creating different class loaders for each version of the library file, thereby saving the processing resources of the above process to the greatest extent.

[0126] It should be emphasized that the class isolation framework adopted in the above process is not specifically limited in this specification, and in actual applications, it can be flexibly selected based on specific requirements.

[0127] In an illustrated embodiment, taking the above real-time computing platform as the Flink platform as an example, the above process can specifically be a JVM process, and the class isolation framework can specifically include the Koupleless framework.

[0128] Among them, Koupleless is a modular Serverless technology solution. In actual applications, it is usually used for application deployment based on the Serverless service mode. It can enable ordinary applications to evolve into Serverless application modules at a relatively low cost, and thus run modularly in the Serverless service mode. Based on the Koupleless framework, class isolation can be achieved within a single JVM process, allowing different Serverless application modules to run independently in the same process without interfering with each other.

[0129] In actual applications, the Koupleless framework usually adopts a modular design, splitting the application program to be deployed into multiple independent Serverless application modules, and by introducing a plug-in mechanism, providing plug-ins for each Serverless application module to expand the functions of the Serverless application module without modifying the code of each Serverless application module.

[0130] Among them, the Serverless application module is responsible for running the core business logic of the application. Each Serverless application module can correspond to a different class loader respectively, ensuring that the classes included in the core business logic run by the Serverless application module will not conflict with other Serverless application modules or plug-ins, thereby achieving class isolation. Each Serverless application module can expand its own functions by loading and calling plug-ins.

[0131] The plug-in is responsible for expanding the functions of the Serverless application module. Each plug-in can also correspond to different class loaders respectively, so that additional function expansion can be provided for the Serverless application module without interfering with the Serverless application module that has loaded the plug-in.

[0132] For example, in an illustrated implementation manner, the Ark (ArkContainer) component can be specifically integrated in the Koupleless framework. The above plug-in can specifically include the Ark-Plugin plug-in created based on the Ark component; the above Serverless application module is specifically the Ark-Biz module created based on the Ark component.

[0133] In this specification, if the JVM process used to execute the compilation task on the Flink platform adopts the koupleless framework as the class isolation framework, on the one hand, after dynamically creating a class loader for the above custom dependencies on which the compilation task corresponding to the obtained compilation request depends, the class loader dynamically created for the custom dependencies can also be abstracted into a Serverless application module for executing the compilation task;

[0134] For example, in practical applications, the class loader dynamically created for the custom dependencies can be further constructed into the form of a Serverless application module for executing the compilation task to complete the abstraction of the class loader. For instance, if the Ark component is integrated in the Koupleless framework, the class loader dynamically created for the custom dependencies can be further constructed into the form of an Ark-Biz module for executing the compilation task by using the Ark component.

[0135] It should be noted that the Serverless application module corresponds to the thread dynamically created for the compilation request. In practical applications, the Serverless application module can be specifically run based on the thread.

[0136] On the other hand, the library files related to various code compilation functions provided by the Flink platform can be abstracted into the form of Ark-Plugin plug-ins and cached in the heap memory of the JVM process for dynamic loading and invocation by the Serverless application module.

[0137] For example, in practical applications, class loaders created separately for library files related to various code compilation functions provided for the Flink platform can be constructed into the form of the above-mentioned plugins respectively to complete the abstraction of the class loader. For example, if the Ark component is integrated into the Koupleless framework, the Ark component can be used to construct the class loaders created separately for library files related to various code compilation functions provided for the Flink platform into the form of Ark-Plugin plugins respectively.

[0138] It should be noted that if the library files related to any one of the code compilation functions provided by the real-time computing platform specifically include multiple versions of library files corresponding to multiple different functional versions of the code compilation function, different class loaders can be created by the Jvm process for these multiple versions of library files respectively and abstracted into different Ark-Plugin plugins respectively.

[0139] In this case, when dynamically loading class loaders corresponding to each of the library files in the at least one library file from the class loaders cached in the memory of the slave process and dynamically creating a class loader for the above-mentioned custom dependencies, specifically, at least one plugin constructed based on the class loaders corresponding to each of the library files in the at least one library file can be dynamically loaded from the plugins cached in the heap memory of the Jvm process.

[0140] When the above-mentioned at least one plugin is loaded, in response to the event of the completion of the loading of the above-mentioned at least one plugin, a class loader can be further dynamically created for the above-mentioned custom dependencies, and a Serverless application module running based on the above-mentioned thread can be further constructed based on the dynamically created class loader to execute the above-mentioned compilation task.

[0141] In the process of executing the above-mentioned compilation task, first, the Serverless application module can be run based on the above-mentioned thread. Among them, since the Serverless application module is a module constructed based on the class loader dynamically created for the above-mentioned custom dependencies, at this time, the classes included in the above-mentioned custom dependencies can be loaded based on the dynamically created class loader.

[0142] Secondly, in the process of running the Serverless application module, the above-mentioned at least one plugin that has been dynamically loaded can be further called. Since the above-mentioned at least one plugin is a plugin constructed based on the class loaders created separately for the above-mentioned at least one library file, at this time, the classes included in each of the above-mentioned at least one library file can be loaded based on the class loaders created separately for the above-mentioned at least one library file.

[0143] After the class loading isolation mechanism supported by the class isolation framework adopted based on the above process has completed the loading of classes included in at least one dependency on which the above compilation task depends, the Serverless application module can execute the loaded classes to complete the compilation of the real-time computing task code included in the compilation request.

[0144] Please refer to Figure 3 , Figure 3 which is a schematic diagram of a three-layer task execution framework based on Koupleless designed for the Jvm process shown in this specification.

[0145] Among them, in this embodiment, it will be described by taking the integration of the Ark component in the Koupleless framework, the above-mentioned plugin is specifically the Ark-Plugin plugin created based on the Ark component, and the above-mentioned Serverless application module is specifically the Ark-Biz module created based on the Ark component as an example.

[0146] As Figure 3 shown, the task execution framework may include a base layer, a Plugin layer, and a Biz layer.

[0147] Among them, the Biz layer is used to maintain the Ark-Biz module constructed based on the class loader dynamically created for the above-mentioned custom dependencies; among them, each Ark-Biz module in the Biz layer corresponds to an independent thread, specifically used to execute the compilation task corresponding to a obtained compilation request.

[0148] The Plugin layer is used to maintain the Ark-Plugin plugin constructed based on the class loader created for the library files related to various code compilation functions provided for the Flink platform; among them, the Ark-Plugin plugin maintained by the Plugin layer can be cached in the heap memory of the JVM process for dynamic loading and calling by the Serverless application module.

[0149] The base layer is used to maintain the library files related to other basic functions other than the code compilation function provided by the Jvm process. For example, the library files maintained by the base layer may specifically include management library files, which are specifically used to manage the Ark-Plugin plugin maintained in the Plugin layer, and will not be listed one by one in this specification.

[0150] Please refer to Figure 4 , Figure 4 which is a flowchart of executing a compilation task based on the Figure 3 task execution framework shown in this specification.

[0151] As Figure 4As shown in the figure, after the server of the Flink platform obtains the compilation request submitted by the user, on the one hand, it can parse the compilation request to determine at least one dependency on which the compilation task corresponding to the real-time computing task code included in the compilation request depends; for example, the above at least one dependency may include all the library files stored in the Flink-lib directory, the library files corresponding to the connector, backend and other plugin packages stored in the Flink-opt directory, the source code submitted by the user corresponding to the real-time computing task code, and the user-defined UDF on which the source code depends.

[0152] On the other hand, it can dynamically create a thread for the compilation request in the Jvm process.

[0153] Please continue to refer to Figure 4 , after determining at least one dependency on which the above compilation task depends, it is possible to dynamically load at least one Ark-Plugin plugin corresponding to the above at least one dependency from the Plugin layer, that is, the Ark-Plugin plugin cached in the heap memory of the JVM process, and determine whether all of the above at least one Ark-Plugin plugin has been successfully loaded. For example, it is possible to determine whether the above at least one Ark-Plugin plugin has been successfully loaded from the heap memory of the JVM process into the stack memory of this thread.

[0154] If the above at least one Ark-Plugin plugin has not been loaded completely, for example, some of the above at least one Ark-Plugin plugin is not cached in the heap memory of the JVM process, at this time the JVM process can dynamically construct this missing part of the Ark-Plugin plugin and also load the dynamically constructed part of the Ark-Plugin plugin into the stack memory of this thread.

[0155] Please continue to refer to Figure 4 , if the above at least one Ark-Plugin plugin has been loaded completely, it is possible to further dynamically create a class loader for the above custom dependencies and further construct an Ark-Biz module running based on this thread based on the dynamically created class loader.

[0156] Then, it is possible to run the Ark-Biz module based on this thread and execute the compilation task corresponding to this compilation request;

[0157] For example, during the process of running the Ark-Biz module based on this thread, on the one hand, it is possible to load the source code submitted by the user corresponding to the real-time computing task code and the classes included in the user-defined UDF on which the source code depends respectively based on the class loader dynamically created for the custom dependencies;

[0158] On the other hand, at least one of the above-mentioned dynamically loaded Ark-Plugin plugins can be called to load, based on the class loaders for building at least one of the above-mentioned Ark-Plugin plugins, all the library files stored in the Flink-lib directory and the classes included in the library files corresponding to the plugin packages such as connectors and backends stored in the Flink-opt directory, respectively.

[0159] After all the classes are loaded, the Ark-Biz module can execute these loaded classes to complete the compilation of the real-time computing task code included in the compilation request. The specific compilation process will not be elaborated in this specification.

[0160] It should be noted that since the Ark-Biz module only selectively and dynamically loads some Ark-Plugin plugins with dependencies from the Plugin layer during the compilation task execution; therefore, in actual applications, those Ark-Plugin plugins maintained in the Plugin layer that have no dependency relationship with the Ark-Biz module can be set to be invisible to the Ark-Biz module, so as to ensure the isolation between the compilation tasks corresponding to different compilation requests to the greatest extent.

[0161] In this specification, in order to make full use of the resources of the Jvm process, a forced recycling mechanism for the threads dynamically created for executing the compilation task can also be introduced into the Jvm process.

[0162] In one shown embodiment, the threads in the above process that are in the idle state can be scanned regularly, and it can be determined whether the scanned threads in the idle state are the threads for executing the above compilation task; for example, it can be confirmed whether the scanned thread in the idle state is the thread for executing the above compilation task by determining whether the class loader corresponding to the scanned idle process is the class loader corresponding to the above Ark-Biz module or Ark-Plugin plugin.

[0163] If the scanned thread in the idle state is the thread for executing the above compilation task, it can be further determined whether the compilation request corresponding to the compilation task executed by this thread is an invalid compilation request; if so, the thread can be recycled.

[0164] Of course, if the scanned thread in the idle state is not the thread for executing the above compilation task, at this time this thread may be a thread reserved for some tasks in the process, and at this time this thread may not be recycled.

[0165] In practical applications, in the off-heap memory of the Jvm process, a metaspace is usually set for the classes loaded during the execution of the compilation task. This metaspace is typically used to cache metadata related to the classes loaded by the process. For example, this metadata can be the class-related metadata contained in at least one of the dependencies loaded by the threads running within the process during the execution of the above compilation task. However, during the continuous execution of the compilation task based on this Jvm process, as the number of loaded classes continues to increase, the metadata cached in this metaspace will also continuously increase. When the data scale of the metadata cached in the metaspace reaches a certain level, it will have a negative impact on the normal operation of the Jvm process.

[0166] Based on this, in order to address the problem of the slow growth of the metadata cached in the off-heap memory of the Jvm process, a mechanism for forcibly restarting the Jvm can also be introduced into the Jvm process.

[0167] In one illustrated embodiment, it is also possible to periodically detect whether the storage space occupied by the metadata cached in the off-heap memory of the Jvm process reaches a threshold; if so, the Jvm process can be forcibly restarted.

[0168] In the above technical solution, when the real-time computing platform executes the compilation task corresponding to the real-time computing task code submitted by the user, on the one hand, by dynamically creating threads for executing the compilation task within the process based on the class isolation framework and executing the compilation task based on the threads within the process, the lighter weight characteristics of the threads can be fully utilized to reduce the resource consumption during the execution of the compilation task and improve the execution speed of the compilation task; on the other hand, during the process of running the threads to load the dependencies required by the compilation task, by using the class loading isolation mechanism supported by the class isolation framework of the process to isolate and load the classes contained in the dependencies, not only can the correct loading of the dependencies be ensured, but also the isolation and security between different compilation tasks can be taken into account.

[0169] Corresponding to the embodiments of the foregoing method, this specification also provides embodiments of an apparatus, an electronic device, and a storage medium.

[0170] Figure 5 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Please refer to Figure 5, at the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510. Of course, it may also include other required hardware. One or more embodiments of this specification can be implemented in a software manner. For example, the processor 502 reads the corresponding computer program from the non-volatile memory 510 into the memory 508 and then runs it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementation manners, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, and can also be hardware or a logic device.

[0171] As Figure 6 shown, Figure 6 is a block diagram of a compilation device for real-time computing task code shown in accordance with an exemplary embodiment of this specification. The device can run in an electronic device as shown in Figure 5 to implement the technical solution of this specification. The electronic device can be specifically applied to a real-time computing platform, and a process for executing a compilation task corresponding to the real-time computing task code submitted by the user runs on the real-time computing platform; wherein, the process is a process based on a class isolation framework; the device 600 includes:

[0172] An acquisition module 601, which acquires a compilation request corresponding to the real-time computing task code submitted by the user;

[0173] An analysis module 602, which analyzes the compilation request, determines at least one dependency on which the compilation task corresponding to the real-time computing task code depends, and dynamically creates a thread for executing the compilation task within the process;

[0174] A loading module 603, which runs the thread, isolates and loads the classes included in the at least one dependency based on the class loading isolation mechanism supported by the class isolation framework, and executes the loaded classes to complete the compilation of the real-time computing task code.

[0175] Correspondingly, this specification also provides an electronic device, which includes a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to implement the steps in all the method flows described above.

[0176] Correspondingly, this specification also provides a computer-readable storage medium, on which executable computer program instructions are stored; wherein, when the instructions are executed by the processor, the steps in all the method flows described above are implemented.

[0177] Correspondingly, this specification also provides a computer program product, on which executable computer program instructions are stored; wherein, when the computer program instructions are executed by a processor, the steps in all the method flows described previously are implemented.

[0178] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software improvements (improvements to method flows). However, with the development of technology, many method flow improvements today can be regarded as direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement to a method flow cannot be implemented using a hardware entity module. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is an integrated circuit whose logic function is determined by a user's programming of the device. Designers can program themselves to "integrate" a digital system onto a single PLD, without having to ask a chip manufacturer to design and fabricate a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compilers used in program development and writing. The original code before compilation also has to be written in a specific programming language, which is called a Hardware Description Language (HDL), and there is not just one type of HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones currently are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that by simply performing a little logical programming on the method flow using the above-mentioned several hardware description languages and programming it into an integrated circuit, it is easy to obtain the hardware circuit that implements the logical method flow.

[0179] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to logically program the method steps to enable the controller to be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. to achieve the same function. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or structures within the hardware component.

[0180] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude that with the development of future computer technologies, the computers for implementing the functions of the above embodiments can be, for example, personal computers, laptop computers, in-vehicle human-machine interaction devices, cellular phones, camera phones, smart phones, personal digital assistants, media players, navigation devices, email devices, game consoles, tablet computers, wearable devices, or any combination of these devices.

[0181] Although one or more embodiments of this specification provide method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way among many orders of step execution and does not represent the only order of execution. When the actual device or terminal product is executed, it may be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment, or even in a distributed data processing environment). The term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, product or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, product or device. Without further limitation, there is no exclusion of additional identical or equivalent elements in the process, method, product or device comprising the said elements. For example, if terms such as first and second are used to denote names, they do not denote any particular order.

[0182] For convenience of description, when describing the above device, it is divided into various modules according to functions and described separately. Of course, when implementing one or more of this specification, the functions of each module can be implemented in the same or multiple software and / or hardware, or the modules implementing the same function can be realized by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings or direct couplings or communication connections shown or discussed with each other may be through some interfaces, and the indirect couplings or communication connections of the devices or units may be in electrical, mechanical or other forms.

[0183] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device generate a device for realizing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0184] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or multiple blocks.

[0185] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are performed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or multiple blocks.

[0186] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0187] The memory may include non-permanent memory in the computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0188] Computer-readable media include permanent and non-permanent, removable and non-removable media that can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0189] Those skilled in the art should understand that one or more embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, one or more embodiments of this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0190] One or more embodiments of this specification can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification can also be practiced in a distributed computing environment, where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0191] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and reference can be made to the corresponding parts of the method embodiments for the relevant content. In the description of this specification, the description of reference terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0192] The above description is only for the embodiments of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included within the scope of the claims.

Claims

1. A method for compiling a real-time computing task code, applied to a real-time computing platform, wherein a process for executing a compilation task corresponding to the real-time computing task code submitted by a user is running on the real-time computing platform; wherein: The process is a process based on a class isolation framework; it includes: Obtaining a compilation request corresponding to the real-time computing task code submitted by the user; Parsing the compilation request, determining at least one dependency item on which the compilation task corresponding to the real-time computing task code depends, and dynamically creating a thread for executing the compilation task within the process; The thread is run, and based on the class loading isolation mechanism supported by the class isolation framework, the class included in the at least one dependency is isolatedly loaded, and the loaded class is executed to complete the compilation of the real-time computing task code.

2. The method as claimed in claim 1, wherein the class loading isolation mechanism supported by the class isolation framework comprises creating different class loaders for each dependency that the code depends on, and using different class loaders to load the classes contained in each dependency that the code depends on, so as to implement a class isolation mechanism for the classes contained in each dependency; Running the thread, and based on the class loading isolation mechanism supported by the class isolation framework, isolatingly loading the class included in the at least one dependency, comprising: Creating a different class loader for each dependency of the at least one dependency; The thread is run to load the classes contained in each of the dependencies respectively based on the class loaders created for each of the dependencies respectively.

3. The method according to claim 2, wherein the process creates different class loaders for library files related to various code compilation functions provided by the real-time computing platform in advance; the class loaders created for library files related to various code compilation functions are cached in the memory of the process; and the at least one dependency includes a user-defined dependency provided by a user; and, at least one library file associated with at least one code compilation function provided by the real-time computing platform; Running the thread, and loading the classes contained in each of the dependencies based on the class loaders created for each of the dependencies, respectively, including: Dynamically loading class loaders corresponding to respective library files in the at least one library file from class loaders cached in the memory of the process, and dynamically creating a class loader for the custom dependency; The thread is run, and based on the dynamically created class loader, the class included in the custom dependency is loaded, and based on the dynamically loaded class loader, the classes included in each library file are loaded respectively.

4. The method according to claim 3, wherein the library files related to any one of the code compilation functions provided by the real-time computing platform include multiple versions of library files corresponding to multiple different functional versions of the code compilation function; the class loader cached in the memory of the process is a common class loader shared by multiple versions of library files corresponding to the same code compilation function; wherein, The general class loader is not configured with a class loading path; Dynamically loading class loaders corresponding to respective library files in the at least one library file from class loaders cached in the memory of the process, comprising: Determine the versions of the various library files; Dynamically load the universal class loaders corresponding to the versions of the respective library files in the at least one library file from the class loaders cached in the memory of the process, and dynamically configure the class loading paths of the respective library files to the corresponding universal class loaders.

5. The method according to claim 3, wherein the at least one library file comprises at least one basic library file and at least one extended library file related to a code compilation function provided by the real-time computing platform; The custom dependencies include: Source code corresponding to the real-time computing task code; and, the user-defined functions on which the source code depends.

6. According to the method as claimed in claim 5, the real-time computing platform includes a Flink platform; the basic library files include full library files stored in the lib directory of the Flink platform; the extended library files include partial library files specified by the user and stored in the opt directory of the Flink platform.

7. The method of claim 6, wherein the process comprises a Jvm process; the class isolation framework comprises a koupleless framework; the memory of the process caches plug-ins constructed based on class loaders created for library files related to each of the code compilation functions; Dynamically loading class loaders corresponding to respective library files in the at least one library file from class loaders cached in the memory of the process, and dynamically creating a class loader for the custom dependency, including: Dynamically load at least one plug-in constructed based on the class loaders corresponding to the respective library files in the at least one library file from the plug-ins cached in the memory of the process; in response to completion of loading of the at least one plug-in, further dynamically create a class loader for the custom dependency, and further construct a Serverless application module running based on the thread based on the dynamically created class loader; wherein the Serverless application module is used to execute the compilation task; Running the thread, loading the classes included in the custom dependency based on the dynamically created class loader, and loading the classes included in each library file based on the dynamically loaded class loader, including: The Serverless application module is run based on the thread to load the classes included in the custom dependencies based on the dynamically created class loader, and in the process of running the Serverless application module, the at least one dynamically loaded plug-in is called to load the classes included in each library file based on the class loaders corresponding to each library file.

8. According to the method as claimed in claim 7, the plug-in includes a plug-in created based on the Ark component in the koupleless framework; the Serverless application module includes a Biz module created based on the Ark component in the koupleless framework.

9. The method of claim 7, further comprising: Periodically scanning threads in an idle state running in the process; Determine whether the scanned thread in an idle state is a thread for executing the compiling task; If the scanned thread in the idle state is a thread for executing the compile task, it is further determined whether the compile request corresponding to the compile task executed by the thread is an invalid compile request; if yes, the thread is recycled.

10. The method according to claim 7, wherein the memory in the process used to cache the class loader created for the at least one library file is the heap memory of the Jvm process; the off-heap memory of the process also caches the metadata generated by the thread running in the process in the process of loading the class included in the at least one dependency; The method further comprises: Regularly detect whether the storage space occupied by the metadata cached in the off-heap memory reaches a threshold; If yes, restart the Jvm process.

11. A compiling device for real-time computing task codes, applied to a real-time computing platform, on which a process for executing a compiling task corresponding to a real-time computing task code submitted by a user is running; wherein: The process is a process based on a class isolation framework; the device comprises: An acquisition module is used to acquire a compilation request corresponding to the real-time computing task code submitted by the user; A parsing module, parsing the compile request, determining at least one dependency item on which the compile task corresponding to the real-time computing task code depends, and dynamically creating a thread for executing the compile task within the process; A loading module runs the thread, and based on the class loading isolation mechanism supported by the class isolation framework, the class contained in the at least one dependency is isolatedly loaded, and the loaded class is executed to complete the compilation of the real-time computing task code.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

14. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images