A running method, device, equipment and storage medium for development files

Abstract

The present application provides a method, apparatus, device, and storage medium for running development files, which are used to solve the problem of low file running efficiency. The method at least includes: receiving a resource association request from a client; wherein, the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: schedulable resources pre-allocated from a resource cluster, and a running environment pool pre-configured in the schedulable resources; based on the resource identifier, scheduling the schedulable resources to create a kernel process, and allocating one running environment in the running environment pool to the kernel process; based on the file identifier, establishing a communication connection with the kernel process; when receiving a file running instruction for the target file, calling the kernel process to run the target file based on the communication connection, obtaining and returning a file running result. By presetting resources, the startup time of the kernel process is reduced, thereby improving the file running efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technologies, and in particular, to a method, apparatus, device, and storage medium for running a development file. Background Art

[0002] With the continuous development of technologies, more and more clients can instruct a server to run a development file to obtain a file running result. For example, in a Jupyter Notebook browser, a development file containing various contents such as code, text, formulas, and charts can be created and edited, and these files are saved in the ipynb (IPython Notebook) format. Thus, the Jupyter Notebook browser can instruct the Jupyter Notebook server to run these development files to implement data exploration, data analysis, machine learning experiments, and the like.

[0003] In related technologies, taking the Jupyter Notebook server as an example, the method for running a development file is generally as follows:

[0004] When receiving a file opening instruction for an ipynb file, the Jupyter Notebook server generates a running environment specification and a required amount of running resources for a kernel process for running the ipynb file based on the ipynb file; the Jupyter Notebook server sends a resource request to the YARN cluster based on the running environment specification and the required amount of running resources. After receiving the resource request, the YARN cluster schedules resources to create a kernel process according to the required amount of running resources, and configures a running environment for the kernel process according to the running environment specification; the Jupyter Notebook server establishes a communication connection between the ipynb file and the kernel process.

[0005] When receiving a file running instruction for this ipynb file, the Jupyter Notebook server invokes the kernel process to run the ipynb file based on the communication connection corresponding to this ipynb file to obtain a file running result.

[0006] However, under related technologies, after opening a development file each time, a series of resource preparation processes need to be performed first to start or restart a kernel process, and then this development file can be run after the communication connection between the server and the kernel process is established. Since the startup or restart process of the kernel process takes a long time, the file running efficiency of the development file is low and cannot meet the current high-efficiency development requirements. Summary of the Invention

[0007] An embodiment of the present application provides a method, device, equipment and storage medium for running development files, which are used to solve the problem of low file running efficiency of development files.

[0008] In the first aspect, a method for running a development file is provided, which is applied to a server and includes:

[0009] Receiving a resource association request from a client; wherein, the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: schedulable resources pre-allocated from a resource cluster, and a running environment pool pre-configured in the schedulable resources;

[0010] Based on the resource identifier, scheduling the schedulable resources to create a kernel process, and allocating a running environment in the running environment pool to the kernel process;

[0011] Based on the file identifier, establishing a communication connection with the kernel process;

[0012] When receiving a file running instruction for the target file, based on the communication connection, calling the kernel process to run the target file, obtaining and returning a file running result.

[0013] In the second aspect, a method for running a development file is provided, which is applied to a resource cluster and includes:

[0014] Receiving a resource scheduling request from a server; wherein, the resource scheduling request is generated by the server based on a resource association request received from a client, and the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: pre-allocated schedulable resources, and a running environment pool pre-configured in the schedulable resources;

[0015] Based on the schedulable resources indicated by the resource identifier carried in the resource scheduling request, creating a kernel process, and allocating a running environment in the running environment pool indicated by the resource identifier to the kernel process;

[0016] When the kernel process receives a process call request carrying the file content of the target file, running the target file based on the file content, obtaining and returning a file running result; wherein, the process call request is generated by the server after establishing a communication connection with the kernel process based on the file identifier and when receiving a file running instruction for the target file, and is sent to the kernel process based on the communication connection.

[0017] In the third aspect, a device for running a development file is provided, which is applied to a server and includes:

[0018] Transmission and reception module: used to receive a resource association request from a client; wherein, the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: schedulable resources pre-allocated from a resource cluster, and a running environment pool pre-configured in the schedulable resources.

[0019] Processing module: used to schedule the schedulable resources to create a kernel process based on the resource identifier, and allocate a running environment from the running environment pool to the kernel process.

[0020] The processing module is further used to: establish a communication connection with the kernel process based on the file identifier.

[0021] The transmission and reception module is further used to: when receiving a file running instruction for the target file, call the kernel process to run the target file based on the communication connection, obtain and return a file running result.

[0022] Optionally, the transmission and reception module is further used to:

[0023] Before receiving the resource association request from the client, receive a resource creation instruction from the client; wherein, the resource creation instruction carries: a cluster identifier of a resource cluster, a schedulable identifier, and at least one running environment specification.

[0024] Based on the cluster identifier, the schedulable identifier, and the at least one running environment specification, send a resource configuration instruction to the resource cluster, so that the resource cluster allocates schedulable resources based on the schedulable identifier carried in the resource configuration instruction, configures a running environment pool in the schedulable resources based on the at least one running environment specification carried in the resource configuration instruction, and returns a resource configuration result.

[0025] The processing module is further used to: determine the resource identifier of the pre-configured resource based on the received resource configuration result.

[0026] Optionally, if a data processing framework is deployed in the resource cluster, the processing module is specifically used to:

[0027] Obtain a data processing policy written in a first development language; wherein, the data processing policy indicates: starting a kernel service, and the kernel service is used to create and call a kernel process.

[0028] Based on the task submission format provided by the data processing framework for the first development language, fuse the data processing policy, the schedulable identifier, and at least one running environment specification to obtain a data processing task.

[0029] Based on the cluster identifier, submit the data processing task to the resource cluster, so that the data processing framework schedules the schedulable resources allocated according to the schedulable identifier in the resource cluster, starts a kernel service, and configures a running environment pool for the kernel service in the schedulable resources according to the at least one running environment specification.

[0030] Optionally, the transceiver module is specifically configured to: receive a resource configuration result; wherein, the resource configuration result includes the network address of the kernel service.

[0031] The processing module is specifically configured to: encode the network address to obtain the resource identifier of the pre-configured resources.

[0032] Optionally, the kernel service is associated with a dependency installation location, and multiple dependency environments are installed on the dependency installation location.

[0033] Each running environment in the running environment pool configured for the kernel service is configured based on at least one dependency environment inherited from the multiple dependency environments.

[0034] Optionally, the transceiver module is further configured to:

[0035] Receive a dependency addition instruction for the running environment pool; wherein, the dependency addition instruction carries the installation package identifier of a dependency environment.

[0036] Based on the network address of the kernel service, send a dependency installation request carrying the installation package identifier to the kernel service, so that the kernel service downloads the environment installation package of the one dependency environment from the dependency environment source based on the installation package identifier, and installs the one dependency environment at the dependency installation location associated with the kernel service based on the environment installation package.

[0037] Optionally, the processing module is specifically configured to:

[0038] Based on the resource identifier, schedule the schedulable resources to perform the following operations:

[0039] Start a kernel service using the main process; wherein, the kernel service is used to create and call kernel processes.

[0040] Call the kernel service, and use the child process to execute: create a kernel process, and based on the respective environment states of the running environments included in the running environment pool, select an idle running environment from the running environment pool and allocate it to the kernel process.

[0041] Optionally, a data processing framework is deployed in the resource cluster, and the kernel process is created by a kernel service started by the data processing framework in the resource cluster;

[0042] Then the processing module is specifically configured to:

[0043] Establish an association relationship between the file identifier and the resource identifier;

[0044] When receiving the network address of the kernel service reported by the kernel service, establish a mapping relationship between the resource identifier and the network address;

[0045] Based on the association relationship, the mapping relationship, and the process identifier assigned by the kernel service to the kernel process, establish a communication connection with the kernel process.

[0046] Optionally, the transceiver module is further configured to:

[0047] Receive a collaboration association request from the client; wherein, the collaboration association request carries: a collaboration identifier of a collaboration file for running and debugging the running process of the target file, and the resource identifier;

[0048] The processing module is further configured to:

[0049] Based on the resource identifier, schedule the schedulable resources to create a collaboration process, and allocate other running environments in the running environment pool except the one running environment to the collaboration process;

[0050] Based on the collaboration identifier, establish a collaboration connection with the collaboration process;

[0051] When receiving a collaboration running instruction for the collaboration file, call the collaboration process to run the collaboration file based on the collaboration connection, and obtain and return a collaboration running result.

[0052] Fourthly, a running device for a development file is provided, which is applied to a resource cluster and includes:

[0053] A transceiver module: configured to receive a resource scheduling request from a server; wherein, the resource scheduling request is generated by the server based on a received resource association request from a client, and the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: pre-allocated schedulable resources, and a running environment pool pre-configured in the schedulable resources;

[0054] Processing module: used to create a kernel process based on the schedulable resources indicated by the resource identifier carried in the resource scheduling request, and allocate one of the running environments in the running environment pool indicated by the resource identifier to the kernel process;

[0055] The processing module is further used for: when the kernel process receives a process call request carrying the file content of the target file, running the target file based on the file content, obtaining and returning a file running result; wherein, the process call request is generated after the server establishes a communication connection with the kernel process based on the file identifier, and when receiving a file running instruction for the target file, and is sent to the kernel process based on the communication connection.

[0056] Optionally, the transceiver module is further used for:

[0057] Before receiving the resource scheduling request from the server, receiving a resource configuration instruction from the server; wherein, the resource configuration instruction is generated by the server based on the cluster identifier, schedulable identifier, and at least one running environment specification of the resource cluster carried in the resource creation instruction received from the client.

[0058] The processing module is further used for:

[0059] Allocating schedulable resources based on the schedulable identifier carried in the resource configuration instruction;

[0060] Configuring a running environment pool in the schedulable resources based on the at least one running environment specification carried in the resource configuration instruction.

[0061] Optionally, a data processing framework is deployed in the resource cluster, and the kernel process is created by a kernel service started by the data processing framework in the resource cluster; the kernel service is associated with a dependency installation location, and multiple dependency environments are installed at the dependency installation location.

[0062] Then the processing module is specifically used for:

[0063] For the at least one running environment specification, respectively perform the following operations:

[0064] Based on one running environment specification, establishing a basic environment from at least one dependency environment inherited from the multiple dependency environments;

[0065] Configuring multiple basic environments for the kernel service in the schedulable resources according to a preset quantity;

[0066] Adding the configured multiple basic environments as multiple running environments to the running environment pool.

[0067] Optionally, the processing module is further configured to:

[0068] Use the kernel service to receive a dependency installation request for the runtime environment pool; wherein, the dependency installation request is generated by the server based on a received dependency addition instruction for the runtime environment pool, and the dependency addition instruction carries an installation package identifier of a dependency environment;

[0069] Use the kernel service to download an environment installation package of the dependency environment from a preset dependency environment source based on the installation package identifier carried in the dependency installation request;

[0070] Based on the downloaded environment installation package, install the dependency environment at a dependency installation location associated with the kernel service.

[0071] Optionally, a data processing framework is deployed in the resource cluster, and the kernel process is created in the resource cluster using the data processing framework;

[0072] Then the processing module is further configured to:

[0073] Establish a call channel for calling a function written in a second programming language used by the data processing framework using a third programming language; wherein, the third programming language is different from the second programming language;

[0074] Then the processing module is specifically configured to:

[0075] Use the kernel process to instruct the data processing framework to run the target file according to the file content based on the call channel, and obtain a file running result.

[0076] In a fifth aspect, a computer program product is provided, including a computer program which, when executed by a processor, implements the method described in the first aspect or the second aspect.

[0077] In a sixth aspect, a computer device is provided, including:

[0078] A memory for storing program instructions;

[0079] A processor for calling the program instructions stored in the memory and executing the method described in the first aspect or the second aspect according to the obtained program instructions, including:

[0080] Receive a resource association request from a client; wherein, the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: schedulable resources pre-allocated from a resource cluster, and a runtime environment pool pre-configured in the schedulable resources;

[0081] Schedule the schedulable resources to create a kernel process based on the resource identifier, and allocate one of the operating environments in the operating environment pool to the kernel process;

[0082] Establish a communication connection with the kernel process based on the file identifier;

[0083] When a file running instruction for the target file is received, call the kernel process to run the target file based on the communication connection, and obtain and return a file running result.

[0084] In a seventh aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause a computer to execute the method described in the first aspect or the second aspect.

[0085] In the embodiments of the present application, pre-configured resources are pre-configured in a resource cluster to achieve resource preheating. When a resource association request carrying a file identifier of a target file and a resource identifier of a pre-configured resource is received, the kernel process can be directly started or restarted based on the pre-configured resource, without having to go through a series of resource preparation operations after selecting the target file, which greatly reduces the time-consuming for starting or restarting the kernel process. Then, when a file running instruction for the target file is received, the kernel process can be efficiently called based on the communication connection with the kernel process to run the target file, and obtain and return a file running result, effectively improving the file running efficiency.

[0086] Furthermore, the pre-configured resources include an operating environment pool pre-configured in the schedulable resources. After creating the kernel process, an operating environment can be allocated to the kernel process, enabling the kernel process to run the target file in an independent operating environment, avoiding the situation where running interference causes errors in the running result, and improving the accuracy of file running.

[0087] Furthermore, the pre-configured resources are resident resources. When the operating environment pool in the pre-configured resources contains multiple operating environments, one kernel process can be started or restarted for each of multiple files based on the pre-configured resources, that is, multiple kernel processes that can run independently can be started or restarted simultaneously in the schedulable resources. On the premise of ensuring the accuracy of file running, the unnecessary resource occupation generated when isolating multiple kernel processes is greatly reduced, improving the file running efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1A FIG. is a schematic diagram of an application scenario of a method for running a development file provided by an embodiment of the present application;

[0089] Figure 1B FIG. is a schematic diagram of the principle of a method for running a development file in the related art;

[0090] Figure 1C Schematic diagram of an application field for the method of running a development file provided by an embodiment of the present application;

[0091] Figure 2 First schematic flow diagram of the method of running a development file provided by an embodiment of the present application;

[0092] Figure 3A First schematic principle diagram of the method of running a development file provided by an embodiment of the present application;

[0093] Figure 3B Schematic principle of the method of running a development file provided by an embodiment of the present application Figure 2 ;

[0094] Figure 4A Schematic flow of the method of running a development file provided by an embodiment of the present application Figure 2 ;

[0095] Figure 4B Third schematic principle diagram of the method of running a development file provided by an embodiment of the present application;

[0096] Figure 4C Fourth schematic principle diagram of the method of running a development file provided by an embodiment of the present application;

[0097] Figure 5A Fifth schematic principle diagram of the method of running a development file provided by an embodiment of the present application;

[0098] Figure 5B Schematic flow of the method of running a development file provided by an embodiment of the present application Figure 2 ;

[0099] Figure 5C Schematic principle of the method of running a development file provided by an embodiment of the present application Figure 6 ;

[0100] Figure 6 Third schematic flow diagram of the method of running a development file provided by an embodiment of the present application;

[0101] Figure 7A Schematic interaction diagram of the device for running a development file provided by an embodiment of the present application;

[0102] Figure 7B Seventh schematic principle diagram of the method of running a development file provided by an embodiment of the present application;

[0103] Figure 8A First schematic structural diagram of the device for running a development file provided by an embodiment of the present application;

[0104] Figure 8B Schematic diagram of a structure of an operating device for a development file provided by an embodiment of the present application Figure 2 ;

[0105] Figure 9 Schematic diagram III of a structure of an operating device for a development file provided by an embodiment of the present application. Detailed implementation manners

[0106] In order to make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

[0107] Some terms in the embodiments of the present application will be explained below to facilitate understanding by those skilled in the art.

[0108] (1) Jupyter Notebook:

[0109] Jupyter Notebook is a web application that can create and share documents containing real-time code, equations, visualizations, and narrative text. Jupyter Notebook is essentially an interactive computing environment mainly used to perform tasks in many fields of data science such as data cleaning, data analysis, and machine learning.

[0110] (2) Enterprise Gateway:

[0111] Enterprise Gateway is a gateway component used in an enterprise-level environment that can be used to connect objects, data analysis tools, and backend computing resources, etc. Enterprise Gateway can coordinate tools such as objects, data analysis tools, and backend computing resources with computing resources in an enterprise-level environment, enabling data processing tasks to be executed efficiently.

[0112] (3) Kernel process:

[0113] The kernel process is a process running in the background used to execute the code input by the front end. Taking IPython Kernel as an example, it is the "execution engine" of Python code. When a Python statement (such as print("Hello, World!")) is entered in a code cell of the Jupyter Notebook browser, IPython Kernel can execute this Python statement and return the execution result to the Jupyter Notebook browser for display.

[0114] (4)Pyenv:

[0115] Pyenv is a tool for managing Python versions, which can install, switch, and use multiple different versions of Python in a system, enabling switching between various Python versions.

[0116] (5)Spark framework and Spark Context:

[0117] The Spark framework is a fast, general-purpose, and scalable distributed data processing framework, mainly used for large-scale data processing and analysis. The Spark framework provides an efficient and flexible computing model that can process various types of data, including structured data (such as tabular data in a database), semi-structured data (such as XML files, JSON files), and unstructured data (such as text, images, videos).

[0118] Spark Context is a core component in the Spark framework, responsible for coordinating and managing all resources and operations related to the Spark framework.

[0119] (6)Java Gateway and JVM Gateway:

[0120] Java Gateway refers to a channel for connecting different Java components, systems, or for interacting between a Java program and an external system in a Java language-related development environment, mainly focusing on connection and communication functions within the scope of Java language technology.

[0121] JVM Gateway is a connection function related to the Java Virtual Machine (JVM). JVM Gateway is a bridge for interacting between the JVM environment and other external environments (which may be programs written in other languages, different computing frameworks, or external services, etc.).

[0122] (7)Monkey Patch:

[0123] Monkey Patch is a technique for dynamically modifying classes or modules at runtime, allowing the behavior of existing classes, functions, or objects to be modified or extended without modifying the original code.

[0124] It should be noted that in the embodiments of this application, data operations such as receiving and sending requests and scheduling resources are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0125] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other relevant parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of that module or unit.

[0126] The application field of the method for running a development file provided by the embodiments of the present application will be briefly introduced below.

[0127] With the continuous development of technology, more and more clients can instruct the server to run a development file to obtain the file running result. For example, in the Jupyter Notebook browser, development files containing various contents such as code, text, formulas, and charts can be created and edited, and these files are saved in the ipynb (IPython Notebook) format. Thus, the Jupyter Notebook browser can instruct the Jupyter Notebook server to run these development files to achieve data exploration, data analysis, machine learning experiments, etc.

[0128] Please refer to Figure 1A , in the Jupyter Notebook browser, there is a development file named "XXXXXX / xxxxxxxx / xxxxx.ipynb". In the "In[1]" cell of this development file, the code "pringt('Hello world!')" can be entered. The Jupyter Notebook browser can instruct the Jupyter Notebook server to run this development file, obtain the file running result "Hello world!", and present this file running result.

[0129] In the related art, taking the Jupyter Notebook server as an example, please refer to Figure 1B , the method for running a development file is generally:

[0130] When receiving a file opening instruction for an ipynb file, the Jupyter Notebook server generates the running environment specifications and the required running resources for the kernel process to run the ipynb file based on the ipynb file; the Jupyter Notebook server sends a resource request to the YARN cluster based on the running environment specifications and the required running resources. After receiving the resource request, the YARN cluster schedules resources to create a kernel process according to the required running resources, and configures the running environment for the kernel process according to the running environment specifications; the YARN cluster registers the kernel process in the Jupyter Notebook server to establish a communication connection between the ipynb file and the kernel process.

[0131] When receiving a file running instruction for this ipynb file, the Jupyter Notebook server calls the kernel process to run the ipynb file based on the communication connection corresponding to this ipynb file, and obtains the file running result.

[0132] However, under the related technology, after opening a development file each time, a series of resource preparation processes need to be passed through before a kernel process can be started or restarted. Furthermore, after the communication connection between the server and the kernel process is established, this development file can be run. Since the startup or restart process of the kernel process takes a long time, the file running efficiency of the development file is low and cannot meet the current high-efficiency development requirements.

[0133] To solve the problem of low file running efficiency of development files, this application proposes a method for running development files. This method is applied to the server. In this method, a resource association request from a client is received. The resource association request carries: the file identifier of the target file and the resource identifier of the pre-configured resources; the pre-configured resources include: schedulable resources pre-allocated from a resource cluster, and a running environment pool pre-configured in the schedulable resources. Based on the resource identifier, the schedulable resources are scheduled to create a kernel process, and a running environment in the running environment pool is allocated to the kernel process. Based on the file identifier, a communication connection with the kernel process is established. When receiving a file running instruction for the target file, the kernel process is called based on the communication connection to run the target file, and the file running result is obtained and returned.

[0134] In the embodiment of the present application, pre-configured resources are pre-configured in the resource cluster to achieve resource preheating. When a resource association request carrying the file identifier of the target file and the resource identifier of the pre-configured resources is received, the kernel process can be directly started or restarted based on the pre-configured resources, without having to go through a series of resource preparation work after the target file is selected, which greatly reduces the time-consuming for starting or restarting the kernel process. Then, when a file running instruction for the target file is received, the kernel process can be efficiently called based on the communication connection with the kernel process to run the target file, and the file running result is obtained and returned, effectively improving the file running efficiency.

[0135] Further, the pre-configured resources include a pre-configured runtime environment pool in the schedulable resources. After the kernel process is created, a runtime environment can be allocated to the kernel process, enabling the kernel process to run the target file in an independent runtime environment, avoiding the situation where the running result is incorrect due to running interference, and improving the file running accuracy.

[0136] Further, the pre-configured resources are resident resources. In the case where the runtime environment pool in the pre-configured resources includes multiple runtime environments, based on the pre-configured resources, a kernel process can be started or restarted for each of multiple files, that is, multiple kernel processes that can run independently can be started or restarted simultaneously in the schedulable resources. On the premise of ensuring the file running accuracy, the unnecessary resource occupation generated when isolating multiple kernel processes is greatly reduced, and the file running efficiency is improved.

[0137] The application scenario of the method for running a development file provided by the present application will be described below.

[0138] Please refer to Figure 1C , which is a schematic diagram of an application scenario of the method for running a development file provided by the present application. This application scenario includes a client 101, a server 102, and a resource cluster 103; communication can be carried out between the client 101 and the server 102, and communication can be carried out between the server 102 and the resource cluster 103; the communication method can be communication using wired communication technology, for example, communication through a connected network cable or serial cable; or it can be communication using wireless communication technology, for example, communication through technologies such as Bluetooth or wireless fidelity (WIFI), and specific details are not limited.

[0139] The client 101 generally refers to devices that can write files, send requests, or receive the running results of files, etc. For example, it includes terminal devices, third-party applications accessible by the terminal device, or web pages accessible by the terminal device, etc. The server 102 generally refers to devices that can receive requests, schedule resources to run files, etc. For example, it includes terminal devices or servers, etc. The resource cluster 103 generally refers to a cluster of devices containing various resources, etc. For example, it includes terminal devices or servers, etc.

[0140] Terminal devices include, but are not limited to, mobile phones, computers, intelligent medical devices, smart home appliances, vehicle-mounted terminals, or aircraft, etc. Servers include, but are not limited to, cloud servers, local servers, or associated third-party servers, etc. The client 101, the server 102, and the resource cluster 103 can all adopt cloud computing to reduce the occupation of local computing resources; similarly, cloud storage can also be adopted to reduce the occupation of local storage resources.

[0141] In response to an associated operation triggered for a target file and pre-configured resources, the client 101 sends a resource association request to the server 102, and the resource association request is used to associate the target file and the pre-configured resources.

[0142] The server 102 receives the resource association request sent by the client 101, and the server 102 starts a kernel process for the target file based on the pre-configured resources in the resource cluster 103. In response to a running operation triggered for the target file, the client 101 sends a file running instruction to the server 102. The server 102 receives the file running instruction, and the server 102 calls the kernel process started for the target file in the resource cluster 103 to run the target file and obtain the file running result.

[0143] The server 102 sends the file running result to the client 101, and the client 101 receives the file running result and presents the file running result.

[0144] The following is based on Figure 1C , from the perspective of the server, a specific introduction to the method for running a development file provided in the embodiment of the present application is given. Please refer to Figure 2 , which is a schematic flowchart of a method for running a development file provided in the embodiment of the present application.

[0145] S201, receive a resource association request from the client.

[0146] The client can be used to create a development file or open an already created development file, and can also be used to write file content in a selected development language in the development file, etc. The development file is a file written in a development language and can execute a function through running.

[0147] At least one resource identifier of each pre-configured resource can also be presented in the client. The pre-configured resources include: schedulable resources pre-allocated from a resource cluster, and a pool of running environments pre-configured in the schedulable resources. The schedulable resources can be resources deployed within a geographical range in a distributed resource cluster; they can also be a certain amount of resources in the resource cluster; or they can be resources divided into the same resource group in the resource cluster, etc., without specific limitations.

[0148] The resource cluster can provide various resources, including computing resources, storage resources, control nodes, execution nodes, etc. The resource cluster can be a yarn cluster, a mesos cluster, a kubernetes cluster, a slurm cluster, etc., without specific limitations. In the embodiments of this application, the yarn cluster is taken as an example for introduction.

[0149] The pool of running environments contains at least one running environment. The running environment can include a development language with at least one installed language version, such as Pyenv; it can also include the dependency environment required to run the file. The dependency environment is, for example, front-end styles, functional functions, developed functional modules, etc., without specific limitations.

[0150] The resource identifier is used to uniquely represent a pre-configured resource. The resource identifier can include at least one of information such as the number of the pre-configured resource, name, resource type of the resource cluster, name of the schedulable resource, and names of the respective running environments in the pool of running environments.

[0151] The resource association request can carry the file identifier of the target file and the resource identifier of the pre-configured resource. The file identifier of the target file can uniquely represent the target file. The file identifier can include at least one of information such as the name, number, and storage location of the target file in the client or the server. The resource association request can be used to indicate associating the target file with the pre-configured resource, so that a kernel process can be started based on the pre-configured resource to run the target file.

[0152] Please refer to Figure 3A , the client presents the file identifiers of multiple files, including file identifier_1, file identifier_2,..., and file identifier_N. The client also presents the resource identifiers of multiple pre-configured resources, including resource identifier_1, resource identifier_2,..., and resource identifier_M. In response to an association operation triggered for file identifier_1 and resource identifier_M, a resource association request is generated. The client sends the resource association request to the server.

[0153] Please refer to Figure 3B, the client presents the file identifiers of multiple files, including File Identifier_1, File Identifier_2, File Identifier_3, File Identifier_4, File Identifier_5, File Identifier_6, File Identifier_7, and File Identifier_8. In response to a file opening operation triggered for File Identifier_1, a file opening instruction is generated. The client sends the file opening instruction to the server, and the server obtains the template content corresponding to the file type based on the file type carried in the file opening instruction. The server returns the template content to the client, and the client receives and presents the template content.

[0154] In response to a selection operation triggered by a resource association button for a target file, the client presents a pop-up window containing the resource identifiers of multiple pre-configured resources. The resource identifiers of the multiple pre-configured resources include Resource Identifier_1, Resource Identifier_2, and Resource Identifier_3. In response to a confirmation selection operation triggered for Resource Identifier_3, a resource association request is generated, and the client sends the resource association request to the server.

[0155] As an embodiment, the pre-configured resources can be pre-configured according to the instructions of the client, so that corresponding pre-configured resources can be configured for personalized scenarios; they can also be pre-automatically configured according to preset configuration policies, so that an efficient configuration process can be achieved through unified standards, etc., and no specific restrictions are made. Resource preheating is achieved through pre-configured resources, so that during the process of starting or restarting a kernel process, the pre-configured resources can be directly used, and a series of resource preparations are no longer required, which can effectively improve the file running efficiency when using the kernel to run files.

[0156] The process of creating pre-configured resources according to the instructions of the client is specifically introduced below. Please refer to Figure 4A .

[0157] S401, receive a resource creation instruction from the client.

[0158] The resource creation instruction carries the cluster identifier, schedulable identifier, and at least one running environment specification of the resource cluster. The cluster identifier is used to uniquely represent the resource cluster. The cluster identifier can be the name, number, etc. of the resource cluster. For example, yarn is used to represent the yarn cluster. The schedulable identifier is used to indicate the resources in the resource cluster that can be divided into a pre-configured resource, such as resources such as control nodes, execution nodes, CPU performance, and storage space. The schedulable identifier can be the name, number, resource quantity, etc. of the schedulable resources, and no specific restrictions are made. The running environment specification can include at least one of the version of the development language, the framework name of the data processing framework, the name of the dependency environment, etc. The running environment specification indicates the dependency environment required to run a kernel process with a certain processing ability.

[0159] Please refer to Figure 4BThe client presents a preconfigured resource list page, which presents a new resource button for creating a preconfigured resource, and also presents multiple created preconfigured resources, and respectively presents the running status, creation time, executable operations, etc. of each preconfigured resource. When creating a preconfigured resource, a creation page of the preconfigured resource can be overlaid and displayed above the preconfigured resource list page. In the creation page, the resource name of the preconfigured resource (such as pyspark), resource description (introduction to the preconfigured resource), etc. can be entered. The resource type of the resource cluster to which the preconfigured resource belongs (such as yarn), the running environment specification (such as spark3.3+python3.6), the schedulable identifier (such as XXXX sub-cluster), the control node (such as Driver) and execution node (such as Executor) corresponding to the schedulable resource amount, and their respective node numbers and resource amounts (such as the number of cores and storage resource amounts) can also be selected.

[0160] S402. Send a resource configuration instruction to the resource cluster based on the cluster identifier, the schedulable identifier, and at least one running environment specification.

[0161] After receiving the resource creation instruction, the resource cluster can be determined based on the resource identifier carried in the resource creation instruction, and a resource configuration instruction can be generated based on the schedulable identifier and at least one running environment specification. The server sends the resource configuration instruction to the resource cluster, and the resource cluster receives the resource configuration instruction sent by the server. The resource cluster allocates schedulable resources based on the schedulable identifier carried in the resource configuration instruction, configures a running environment pool in the schedulable resources based on at least one running environment specification carried in the resource configuration instruction, and returns a resource configuration result. The resource configuration result can indicate whether the configuration is successful, or can indicate the scheduling address of the preconfigured resource. Through the scheduling address, the preconfigured resource can be connected, so that the schedulable resources in the preconfigured resource can be scheduled, and the kernel process can be started or restarted using the running environment, etc.

[0162] As an embodiment, there are multiple methods for sending a resource configuration instruction to the resource cluster. For example, the generated message can be used as the resource configuration instruction by integrating the schedulable identifier and at least one running environment specification based on the network protocol, and the resource configuration instruction can be sent to the transceiver node based on the Internet Protocol (IP) address or port address of the transceiver node in the resource cluster.

[0163] For another example, when a data processing framework is deployed in the resource cluster, the resource configuration instruction can also be sent based on the task submission function provided by the data processing framework, which will be specifically introduced below.

[0164] The data processing framework is used to provide high-performance data processing capabilities, such as the Spark framework, the Hadoop framework, the Flink framework, the Storm framework, etc. There is no specific limitation. In the embodiments of the present application, the Spark framework is taken as an example for introduction.

[0165] Obtain a data processing policy written in a first programming language, where the data processing policy instructs to start a kernel service, and the kernel service is used to create and call kernel processes. The first programming language and the second programming language used by the data processing framework may be the same or different.

[0166] The kernel service can be a computing engine with the ability to perform data processing based on a pre-configured resource. The kernel service can create and call a kernel process based on a pre-configured resource, or can create and call multiple kernel processes based on a pre-configured resource. Thus, the kernel service can flexibly control the usage of the pre-configured resource, enabling multiple kernel processes to share the pre-configured resource and improving resource utilization.

[0167] Based on the task submission format provided by the data processing framework for the first programming language, integrate the data processing policy, the schedulable identifier, and at least one running environment specification to obtain a data processing task. There are various integration methods, such as arranging according to a specified format, filling in a preset template, or setting it in a variable in the form of key-value pairs. Taking the form that can be recognized by the data processing framework as the task form, then the data processing policy, the schedulable identifier, and at least one running environment specification can be converted into a task form that can be recognized by the data processing framework. Thus, relying on the high-performance data processing capabilities of the data processing framework, a kernel service can be started to create and call kernel processes. Regardless of what programming language the data processing policy is written in, the data processing logic therein can be executed in the resource cluster with the help of the data processing framework.

[0168] Based on the cluster identifier, submit the data processing task to the resource cluster, so that the resource cluster uses the data processing framework to schedule the schedulable resources allocated according to the schedulable identifier, start a kernel service, and configure a running environment pool for the kernel service in the schedulable resources according to at least one running environment specification.

[0169] Please refer to Figure 4C, taking the server as a Jupyter Notebook server, the resource cluster as a yarn cluster, and the data processing framework as a Spark framework as an example, the Jupyter Notebook server obtains the pre-stored java program, that is, the data processing strategy, and receives the resource creation instruction sent by the client; the Jupyter Notebook server encapsulates and processes this java program, the schedulable identifier carried by the resource creation instruction, and at least one running environment specification based on the Spark submit task submission format provided by the Spark framework, etc., to obtain a Spark task.

[0170] Submit the Spark task to the yarn cluster indicated by the cluster identifier carried by the resource creation instruction. In the yarn cluster, use the Java entrypoint functional component as the entry point of the Java program, and create a SparkContext as the entry and core control node of the Spark Application according to this java program, which is responsible for underlying functions such as establishing a connection with the cluster, resource allocation and management, and task scheduling. SparkContext will build functional components in the Spark framework such as DAGScheduler and TaskScheduler for subsequent task scheduling and execution.

[0171] After the SparkContext is created, the kernel service can be launched in the Spark Application based on the schedulable identifier. After the kernel service starts, it can report its own ip:port as the network address to the Jupyter Notebook server to establish a communication channel between the Jupyter Notebook server and the kernel service. After the kernel service starts, it can also configure a running environment pool for the kernel service based on at least one running environment specification to complete the creation of pre-configured resources.

[0172] S403, determine the resource identifier of the pre-configured resource based on the received resource configuration result.

[0173] When the resource configuration result indicates successful configuration, generate the resource identifier of the pre-configured resource; when the resource configuration result indicates failed configuration, generate a reconfiguration prompt for the pre-configured resource to indicate reconfiguring the pre-configured resource.

[0174] When a data processing framework is deployed in a resource cluster, the kernel process can be created by a kernel service started by the data processing framework based on a data processing policy. Then, the resource configuration result received upon successful configuration may further include the network address of the kernel service (including IP address, port address, etc.), which is connected to the pre-configured resource and can thus be used as the scheduling address introduced above. When determining the resource identifier of the pre-configured resource, the network address can be encoded, and the encoded network address can be used as the resource identifier of the pre-configured resource. While uniquely representing the pre-configured resource, the mapping relationship between the resource identifier and the network address can also be recorded in the form of the encoded network address. When connecting to the pre-configured resource, the network address of the pre-configured resource can be obtained by decoding the resource identifier, reducing unnecessary occupation of storage resources and helping to improve the utilization rate of storage resources. When determining the resource identifier of the pre-configured resource, the creation sequence number of the pre-configured resource can also be used as the resource identifier to simplify the determination process. Then, in order to connect to the pre-configured resource, the mapping relationship between the resource identifier and the network address can be stored.

[0175] Please refer to Figure 5A , taking the server as a Jupyter Notebook server, the resource cluster as a yarn cluster, the data processing framework as a Spark framework, and the communication between the Jupyter Notebook server and the yarn cluster through an enterprise gateway (such as Enterprise Gateway) as an example. Through the network address reported by the kernel service, the Jupyter Notebook server can send a process start instruction to the kernel service to start kernel process A; it can also send a process restart instruction to the kernel service to restart kernel process B; it can also send a process interruption instruction to the kernel service to pause the running process in kernel process C; it can also send a process end instruction to the kernel service to shut down kernel process D.

[0176] As an embodiment, when a data processing framework is deployed in a resource cluster, the kernel service can be associated with a dependency installation location, which is used to install various dependency environments. The kernel service can update, delete the various dependency environments installed at the dependency installation location, or install new dependency environments at the dependency installation location, thereby enabling unified management of the runtime environment pool.

[0177] The various dependency environments installed at the dependency installation location can form a basic and general operating environment for the kernel processes that start kernel services. Each operating environment in the operating environment pool configured for kernel services is configured based on at least one dependency environment inherited from multiple dependency environments. Therefore, the operating environments in the operating environment pool share some software packages and libraries installed at the dependency installation location. The various dependency environments installed at the dependency installation location will not be deleted or uninstalled when the kernel process shuts down, and the kernel process can continue to inherit the dependency environments therein when it restarts.

[0178] For example, some common data analysis libraries such as NumPy, Pandas, Matplotlib, etc. can be installed at the dependency installation location. These data analysis libraries are used in many data processing and analysis processes. Therefore, the operating environments in the operating environment pool configured for kernel services can inherit these basic dependency environments without reinstalling.

[0179] Therefore, multiple kernel processes created by a kernel service can be based on a unified variety of dependency environments, avoiding the situation where multiple kernel processes cannot cooperate to run files due to different versions or types of installed dependency environments, and improving the running stability of files.

[0180] After the operating environment is allocated to the kernel process, other dependency environments can also be installed separately in this operating environment to achieve a personalized operating environment. For example, the kernel process receives a dependency installation package sent by the client, and the kernel process installs the corresponding other dependency environments in its operating environment based on the received dependency installation package. Installing other dependency environments separately in the operating environment of the kernel process will be deleted or uninstalled after the kernel process shuts down and needs to be reinstalled when the kernel process restarts, so that some infrequently used dependency environments do not always occupy the resources at the dependency installation location, which can improve resource utilization.

[0181] Please refer to Figure 5B , continuing with the resource cluster as the yarn cluster and the data processing framework as the Spark framework as an example, the dependency environment can be inherited from the dependency installation location (such as denoted as compute Pyenv), and the operating environment pool (such as the operating environment is Pyenv) can be maintained for the kernel service in the background. Then the kernel service can reserve a specified number of idle operating environments in the operating environment pool, such as 5 Pyenvs, namely Pyenv_1, Pyenv_2, Pyenv_3, Pyenv_4, and Pyenv_5. When the number of idle operating environments in the operating environment pool does not reach the specified number, new operating environments can be created in the schedulable resources based on at least one operating environment specification. When a kernel process is destroyed, the operating environment used by this kernel process is returned to the operating environment pool.

[0182] As an example, when updating various dependency environments installed at the dependency installation location, taking the addition of a new dependency to the dependency installation location as an example, a new dependency instruction for the runtime environment pool is received. The new dependency instruction can come from a client, the background, or other components, etc., and there is no specific limitation. The new dependency instruction carries the installation package identifier of a dependency environment. Based on the network address of the kernel service, a dependency installation request carrying the installation package identifier is sent to the kernel service, so that the kernel service can download the environment installation package of this dependency environment from the dependency environment source based on the installation package identifier, and install a dependency environment at the dependency installation location associated with the kernel service based on the environment installation package.

[0183] Please refer to Figure 5C , the kernel service uses the pip install command to download the environment installation package from the Python Package Index (PyPI) or other specified sources, and installs the dependency environment at the dependency installation location associated with the kernel service based on the downloaded environment installation package. If you want to install a dependency environment named numpy, then you can execute the pip install numpy command to implement the installation of this dependency environment.

[0184] After updating various dependency environments at the dependency installation location, the updated various dependency environments can be packaged. Taking the Python development environment as an example, this package may include the Python interpreter, the installed various dependency environments, and the configuration information of each dependency environment. The packaging form can be a compressed file or a specific format, such as the tar.gz file format, for transmission and deployment in a distributed environment. After packaging, according to the data transmission method of the data processing framework in the resource cluster, for example, using the sc.add(·) method written by the sparkcontext functional component provided by the JVM Gateway to call the data processing framework as the data transmission method in the resource cluster, the package is distributed to schedulable resources, such as each execution node (such as including execution node_1, execution node_2,..., execution node_N) in the schedulable resources, so that the execution node can maintain the runtime environment pool by inheriting the various dependency environments in the package, enabling the kernel processes started by the kernel service to use the same dependency environment, realizing the sharing of dependency environments at the kernel service level, simplifying the process of installing dependency environments separately for each kernel process, and improving the efficiency and consistency of file operation.

[0185] S202, based on the resource identifier, schedule the schedulable resources to create a kernel process, and allocate a runtime environment in the runtime environment pool to the kernel process.

[0186] The resource identifier carried by the resource association request can be connected to the corresponding pre-configured resource, so that the schedulable resources in the pre-configured resource can be scheduled to create a kernel process, and a running environment can be selected from the running environment pool configured in the schedulable resources and assigned to the kernel process to complete the startup or restart of the kernel process.

[0187] When selecting a running environment from the running environment pool, a running environment can be selected according to the configuration order of each running environment; a running environment with an idle state can also be selected based on the respective environment states of each running environment; a running environment with an environment priority greater than the preset priority can also be selected according to the respective environment priorities of each running environment. The longer the running environment is in the idle state, the higher the environment priority, or the more dependent environments the running environment contains, the higher the environment priority, etc., and there is no specific limitation.

[0188] As an embodiment, when creating a kernel process and allocating a running environment, the schedulable resources can be scheduled to perform the following operations based on the resource identifier:

[0189] Use the main process to start a kernel service, which is used to create and call the kernel process. Call the kernel service and execute it using a subprocess (such as subprocess): create a kernel process, and select an idle running environment from the running environment pool based on the respective environment states of the running environments included in the running environment pool and assign it to the kernel process.

[0190] Since the subprocess has an independent memory space and running space and is isolated from the main process and other subprocesses, when the kernel process starts or restarts in the subprocess, it will not be interfered by other kernel processes or the kernel service running in the main process. The internal running state, variables, and resource allocation of the kernel process are independent of other kernel processes, avoiding abnormal situations such as running conflicts and errors caused by sharing dependent environments and schedulable resources, and improving the running stability of the file.

[0191] Furthermore, specific permissions and security contexts can be assigned to the subprocess when it is created to limit its access permissions to specific resources and sensitive data in the resource cluster, thereby effectively preventing abnormal behaviors of the code running in the kernel process from obtaining specific resources and sensitive data in the resource cluster due to vulnerabilities, etc., and ensuring the data security during file running.

[0192] S203, establish a communication connection with the kernel process based on the file identifier.

[0193] Based on the file identifier, a communication connection can be established between the server and the kernel process for the target file, so that the kernel process can be instructed to run the target file through the communication connection. The communication connection is used to send data to the kernel process and receive data from the kernel process, etc. For example, the communication connection includes the scheduling address of the preconfigured resources introduced above, and the process identifier of the kernel process, etc.; for another example, the communication connection includes the network address of the kernel service introduced above (such as including the IP address or port address, etc.), and the process identifier of the kernel process, etc., and no specific limitation is made.

[0194] As an embodiment, when a data processing framework is deployed in the resource cluster and the kernel process is created by the kernel service started by the data processing framework in the resource cluster, the target file can be associated with the preconfigured resources by establishing an association relationship between the file identifier and the resource identifier; further, when receiving the network address of the kernel service reported by the kernel service, a mapping relationship between the resource identifier and the network address can be established to register the kernel service on the server; a mapping relationship between the network address and the process identifier assigned by the kernel service to the kernel process can also be further established to register the kernel process on the server, etc. The communication connection established with the kernel process based on the association relationship, the mapping relationship, and the process identifier assigned by the kernel service to the kernel process can connect the target file with the kernel process, so that the file content in the target file can be directly sent to the kernel process for running, improving the file running efficiency.

[0195] An enterprise-level gateway (such as Enterprise Gateway) can also be set between the server and the resource cluster. Then, a communication connection can be established between the enterprise-level gateway and the kernel process, which helps to achieve efficient file running.

[0196] S204, when receiving a file running instruction for the target file, call the kernel process to run the target file based on the communication connection, and obtain and return the file running result.

[0197] After associating the target file with the preconfigured resources and starting the kernel process during the association process, if a file running instruction for the target file is received, then the kernel process can be called to run the target file based on the communication connection with the kernel process, and the file running result can be obtained and returned.

[0198] The file running instruction can be generated in response to a file running operation triggered for the client, or can be generated when periodically detecting the file content of the target file and determining that the file content conforms to the running format, etc., and no specific limitation is made. Among them, the running format indicates that there are no syntax errors and no function call relationships that cannot be called in the file content.

[0199] As an embodiment, when a pre-configured resource is associated with multiple files, multiple kernel processes are correspondingly started in the pre-configured resource. Therefore, when multiple kernel processes use schedulable resources to run the respective received files, abnormal situations such as resource scheduling conflicts or resource competition may occur. Therefore, a third programming language is used to pre-establish a call channel for a function implemented in a second programming language, so that at the level of the third programming language, a function call implemented in the third programming language corresponding to the function implemented in the second programming language can be created. Each function call can serve as a call channel for calling the function implemented in the second programming language using the third programming language. The third programming language can be the same as or different from the first programming language; the third programming language is different from the second programming language.

[0200] At the level of the third programming language, unified management, configuration, and restriction can be performed on each function call, so that when the function implemented in the second programming language is called through each function call, abnormal situations such as resource scheduling conflicts or resource competition will not occur, achieving the effect of sharing the function implemented in the second programming language among multiple kernel processes, and effectively ensuring the stability of file operation.

[0201] When a data processing framework using a second programming language is deployed in a resource cluster, the kernel process is created by the data processing framework in the resource cluster. If a call channel for calling the function implemented in the second programming language used by the data processing framework using the third programming language is set in the resource cluster, then after the file content included in the target file is sent to the kernel process based on a communication connection, the kernel process uses the call channel to instruct the data processing framework to run the target file according to the file content and obtain a file operation result.

[0202] For example, when the data processing framework is the Spark framework, the second programming language used by the Spark framework is Python, and the Spark framework can divide pre-configured resources in the resource cluster as a Spark application. Since a Spark application can only initialize a Spark Context once and is responsible for resource application, resource allocation, and resource management of schedulable resources in the entire Spark application, when multiple kernel processes are started based on a pre-configured resource, the Spark Context cannot be shared and used among multiple kernel processes.

[0203] The third development language can be Java. Creating a Spark Context at the Java level can better manage and allocate pre-configured resources. By creating a Spark Context at the Java level through the Java Gateway, the call function can be obtained, and the call function is used to call the Spark Context for data processing. Various parameters of the Spark Context, such as the application name, running mode, resource allocation, etc., can be centrally set in the Java code, without the need to configure them separately in different places, thus improving the consistency and maintainability of the configuration.

[0204] When the Spark Context is created at the Java level, that is, after the call function is obtained, the kernel process can access the call function through the JVM Gateway to achieve the purpose of calling the Spark Context. In this way, the kernel process can obtain the resource information of the Spark cluster, submit tasks, obtain calculation results, etc.

[0205] As an embodiment, the Spark Context created at the Java level can be modified and wrapped through Monkey Patch, so that the way of using the Spark Context in Python is consistent with the way of using it in the original Java or Scala environment. Even though the interaction with the Spark cluster is implemented through the Java Gateway and the JVM Gateway at the bottom layer, in the Python environment, it can be used like using Spark in a pure Python environment, greatly reducing the usage threshold of the call channel and improving the development efficiency.

[0206] For example, in Python, directly call the SparkContext created at the Java level according to the call method of the SparkContext, such as calling the method of creating a Resilient Distributed Dataset (RDD), calling the method of executing a Structured Query Language (SQL) query, etc. These methods can be forwarded to the SparkContext at the Java level through call channels such as the Java Gateway and the JVM Gateway at the bottom layer for execution.

[0207] As an embodiment, in a scenario where multiple files are collaboratively run, multiple kernel processes can be started in a pre-configured resource, and each kernel process runs one file. Thus, multiple files can run based on a pre-configured resource, and multiple files can collaboratively run without resource interference or resource competition, and without abnormal situations such as code chaos.

[0208] Taking the debugging scenario as an example, a collaboration association request from a client is received. The collaboration association request carries a collaboration identifier of a collaboration file for running and debugging the running process of a target file, and a resource identifier of a pre-configured resource. Based on the resource identifier, a schedulable resource is scheduled to create a kernel process as a collaboration process, and other running environments in the running environment pool except the running environment for running the target file are assigned to the collaboration process. Based on the collaboration identifier, a collaboration connection with the collaboration process is established. When a collaboration running instruction for a collaborative file is received, the collaboration process is called based on the collaboration connection to run the collaborative file, and the collaboration running result is obtained and returned.

[0209] The target file and the collaboration file can each run in an independent running environment, and there is no need to associate the target file and the collaboration file in the same kernel process, avoiding the situation of code context chaos and improving the running stability of the file.

[0210] Based on the same inventive concept, from the perspective of a resource cluster, the running method of the development file provided by the embodiments of the present application is specifically introduced. Please refer to Figure 6 , which is a schematic flowchart of a running method of a development file provided by an embodiment of the present application.

[0211] S601, Receive a resource scheduling request from a server.

[0212] The resource scheduling request is generated by the server based on a received resource association request from a client. The resource association request carries a file identifier of a target file and a resource identifier of a pre-configured resource. The pre-configured resource includes pre-allocated schedulable resources and a running environment pool pre-configured in the schedulable resources. For specific details, reference can be made to the previous introduction and will not be elaborated here.

[0213] As an embodiment, before receiving a resource scheduling request from a server, a resource configuration instruction from the server is received. The resource configuration instruction is generated by the server based on a cluster identifier, a schedulable identifier, and at least one running environment specification carried in a resource creation instruction received from a client. Schedulable resources are allocated based on the schedulable identifier carried in the resource configuration instruction. Based on at least one running environment specification carried in the resource configuration instruction, a running environment pool is configured in the schedulable resources. For specific details, reference can be made to the previous introduction and will not be elaborated here.

[0214] As an embodiment, a data processing framework is deployed in a resource cluster, and a kernel process is created by a kernel service started by the data processing framework in the resource cluster. The kernel service is associated with a dependency installation location, and multiple dependency environments are installed at the dependency installation location. For details, please refer to the previous introduction and will not be elaborated here. For at least one running environment specification, the following operations are performed respectively:

[0215] Based on one running environment specification, at least one dependency environment inherited from multiple dependency environments is used to establish a basic environment. According to a preset quantity, multiple basic environments are configured for the kernel service in schedulable resources. The multiple basic environments are added to a running environment pool as multiple running environments. The same dependency environment is included in the multiple basic environments. The running environment pool is maintained in the background, and the number of running environments in the idle state in the running environment pool can be kept at the preset quantity, avoiding the situation that the entire running environment pool is occupied, and improving the running stability of files.

[0216] As an embodiment, a kernel service is used to receive a dependency installation request for the running environment pool. The dependency installation request is generated by the server based on a received dependency addition instruction for the running environment pool, and the dependency addition instruction carries an installation package identifier of a dependency environment. The kernel service is used to download an environment installation package of a dependency environment from a preset dependency environment source based on the installation package identifier carried in the dependency installation request. Based on the downloaded environment installation package, a dependency environment is installed at the dependency installation location associated with the kernel service. For details, please refer to the previous introduction and will not be elaborated here.

[0217] S602, a kernel process is created based on the schedulable resources indicated by the resource identifier carried in the resource scheduling request, and a running environment in the running environment pool indicated by the resource identifier is allocated to the kernel process. For details, please refer to the previous introduction and will not be elaborated here.

[0218] S603, when the kernel process receives a process call request carrying the file content of the target file, the target file is run based on the file content, and a file running result is obtained and returned.

[0219] The process call request is generated by the server after establishing a communication connection with the kernel process based on the file identifier and receiving a file running instruction for the target file, and is sent to the kernel process based on the communication connection. For details, please refer to the previous introduction and will not be elaborated here.

[0220] As an embodiment, a data processing framework is deployed in a resource cluster, and a kernel process is created in the resource cluster by using the data processing framework. A call channel for calling a function written in a second programming language used by the data processing framework is established by using a third programming language. The third programming language is different from the second programming language. The kernel process is used to instruct the data processing framework to run a target file according to the file content based on the call channel to obtain a file operation result. For details, please refer to the previous introduction and will not be elaborated here.

[0221] Taking the server as a Jupyter Notebook server, the resource cluster as a yarn cluster, the data processing framework as a Spark framework, the Jupyter Notebook server and the yarn cluster communicate through an enterprise gateway (DataLab Enterprise Gateway), and the client and the yarn cluster also communicate through this enterprise gateway (DEG) as an example, the running method of the development file provided by the embodiments of the present application will be introduced by way of example. Please refer to Figure 7A and Figure 7B 。

[0222] S701, in response to a resource creation operation triggered by a client, generate a resource creation instruction.

[0223] The resource creation operation includes an operation of inputting the cluster identifier of the resource cluster, an operation of inputting a schedulable identifier, and an operation of inputting at least one running environment specification, etc. Then the resource creation instruction can carry the cluster identifier of the yarn cluster, the schedulable identifier specifying a certain resource pool, and the running environment specification of Spark 3.3 version combined with Python 3.6, etc.

[0224] The client sends a resource creation instruction to the Jupyter Notebook server, and the Jupyter Notebook server receives the resource creation instruction through the enterprise gateway (DEG).

[0225] S702, the enterprise gateway (DEG) obtains a java program recording a data processing policy, and combines the data processing policy, the schedulable identifier, and at least one running environment specification in the task submission format of the Spark framework to obtain a Spark task, and submits the Spark task to the yarn cluster.

[0226] S703, in the yarn cluster, based on the data processing policy indicated by the Spark task, in the schedulable resources indicated by the Spark task, use java Gateway to create a Spark Context at the java level, and pull up a kernel service (Kernel Operator) based on the Spark Context.

[0227] S704, The yarn cluster sends the network address of the kernel service to the enterprise-level gateway (DEG). The Jupyter Notebook server receives the network address of the kernel service sent by the yarn cluster through the enterprise-level gateway (DEG) and establishes a mapping relationship between the resource identifier and the network address.

[0228] S705, In response to a resource association operation triggered by the resource identifier of a target file and a pre-configured resource, a resource association request is generated. For example, first, after the client responds to the file opening operation of the target file, the file content of the target file is presented on the client; then, in the file content, in response to the resource association operation triggered by the resource identifier of a pre-configured resource, a resource association request is generated.

[0229] The client sends a resource association request to the Jupyter Notebook server, and the Jupyter Notebook server receives the resource association request through the enterprise-level gateway (DEG).

[0230] S706, The Jupyter Notebook server determines the network address corresponding to a resource identifier based on the mapping relationship through the enterprise-level gateway (DEG), and sends a process start instruction to the kernel service running in the yarn cluster based on the network address.

[0231] S707, The kernel service creates a kernel process in the schedulable resources included in the pre-configured resource, and allocates a running environment A for the kernel process from the associated running environment pool to complete the startup of the kernel process A.

[0232] S708, The kernel process obtains functions such as SparkContext and Spark Session created at the Java level using the Java Gateway through the JVM Gateway to complete the establishment of the communication connection between the Jupyter Notebook server and the kernel process.

[0233] S709, When receiving the file running instruction A for the target file, the target file is called to run based on the communication connection, and the file running result is obtained and returned.

[0234] S709A: The client generates a file running instruction A in response to the file running operation triggered by the target file; the client sends the file running instruction A to the Jupyter Notebook server through the enterprise-level gateway (DEG).

[0235] S709B: The Jupyter Notebook server sends the target file to the kernel process A started by the YARN cluster through the communication connection to instruct the YARN cluster to call the kernel process A to run the target file.

[0236] S709C: After receiving the target file through the communication connection, the yarn cluster calls kernel process A to run the target file and obtains the file running result.

[0237] S709D: The YARN cluster returns the file running result to the Jupyter Notebook server through the communication connection, and the Jupyter Notebook server receives the file running result sent by the YARN cluster through the communication connection.

[0238] S709E: The Jupyter Notebook server returns the received file running result to the client, and the client receives the file running result sent by the Jupyter Notebook server.

[0239] For collaborative files, the above process is repeated, and the kernel service creates a kernel process in the schedulable resources included in the preconfigured resources, and allocates an operating environment B to the kernel process from the associated operating environment pool, completing the startup of kernel process B. When receiving the file running instruction B for the collaborative file, the kernel process is called based on the communication connection to run the target file, and the file running result is obtained and returned.

[0240] In an embodiment of the present application, the startup time of the kernel process is shortened by pre-configuring resources to be resident. Compared with the effect of starting the kernel process of at least 1 minute obtained by the traditional method of running development files, the embodiment of the present application can complete the startup of the kernel process in 2 seconds, greatly shortening the startup process of the kernel process.

[0241] Furthermore, starting a spark application with a pre-configured resource can start multiple kernel processes in the same spark application, avoiding the situation where a kernel process monopolizes the spark application and causing resource waste. Furthermore, unified dependency package management is performed with a pre-configured resource as a unit, so that the newly added dependency environment can take effect in distributed computing, and the restart of the kernel process will not cause the loss of the dependency environment at the spark application level, etc., which improves the stability of file operation.

[0242] Based on the same inventive concept, the embodiment of the present application provides a development file operation device, which can realize the functions corresponding to the aforementioned development file operation method from the perspective of the server. Figure 8A, the device includes a transceiver module 81 and a processing module 82, where:

[0243] The transceiver module 81: is used to receive a resource association request from the client; where the resource association request carries: a file identifier of the target file and a resource identifier of the pre-configured resource; the pre-configured resource includes: schedulable resources pre-allocated from the resource cluster, and a running environment pool pre-configured in the schedulable resources;

[0244] The processing module 82: is used to schedule the schedulable resources to create a kernel process based on the resource identifier, and allocate a running environment from the running environment pool to the kernel process;

[0245] The processing module 82 is further used to: establish a communication connection with the kernel process based on the file identifier;

[0246] The transceiver module 81 is further used to: when receiving a file running instruction for the target file, call the kernel process to run the target file based on the communication connection, obtain and return the file running result.

[0247] In a possible embodiment, the transceiver module 81 is further used to:

[0248] Before receiving the resource association request from the client, receive a resource creation instruction from the client; where the resource creation instruction carries: a cluster identifier of the resource cluster, a schedulable identifier, and at least one running environment specification;

[0249] Based on the cluster identifier, the schedulable identifier, and at least one running environment specification, send a resource configuration instruction to the resource cluster, so that the resource cluster allocates schedulable resources based on the schedulable identifier carried in the resource configuration instruction, configures a running environment pool in the schedulable resources based on at least one running environment specification carried in the resource configuration instruction, and returns a resource configuration result;

[0250] The processing module 82 is further used to: determine the resource identifier of the pre-configured resource based on the received resource configuration result.

[0251] In a possible embodiment, if a data processing framework is deployed in the resource cluster, the processing module 82 is specifically used to:

[0252] Obtain a data processing policy written in a first development language; where the data processing policy indicates: starting a kernel service, and the kernel service is used to create and call a kernel process;

[0253] Based on the task submission format provided by the data processing framework for the first development language, fuse the data processing policy, the schedulable identifier, and at least one running environment specification to obtain a data processing task;

[0254] Based on the cluster identifier, submit a data processing task to the resource cluster, so that the data processing framework schedules the schedulable resources allocated according to the schedulable identifier in the resource cluster, starts a kernel service, and configures a runtime environment pool for the kernel service in the schedulable resources according to at least one runtime environment specification.

[0255] In one possible embodiment, the transceiver module 81 is specifically configured to: receive a resource configuration result; wherein, the resource configuration result includes the network address of the kernel service.

[0256] The processing module 82 is specifically configured to: encode the network address to obtain the resource identifier of the pre-configured resources.

[0257] In one possible embodiment, the kernel service is associated with a dependency installation location, and multiple dependency environments are installed at the dependency installation location.

[0258] Then each runtime environment in the runtime environment pool configured for the kernel service is configured based on at least one dependency environment inherited from the multiple dependency environments.

[0259] In one possible embodiment, the transceiver module 81 is further configured to:

[0260] Receive a dependency addition instruction for the runtime environment pool; wherein, the dependency addition instruction carries the installation package identifier of a dependency environment.

[0261] Based on the network address of the kernel service, send a dependency installation request carrying the installation package identifier to the kernel service, so that the kernel service downloads the environment installation package of a dependency environment from the dependency environment source based on the installation package identifier, and installs a dependency environment at the dependency installation location associated with the kernel service based on the environment installation package.

[0262] In one possible embodiment, the processing module 82 is specifically configured to:

[0263] Based on the resource identifier, schedule the schedulable resources to perform the following operations:

[0264] Start a kernel service using the main process; wherein, the kernel service is used to create and call kernel processes.

[0265] Call the kernel service and use the child process to execute: create a kernel process, and based on the respective environment states of the runtime environments included in the runtime environment pool, select a runtime environment with an idle environment state from the runtime environment pool and allocate it to the kernel process.

[0266] In one possible embodiment, a data processing framework is deployed in the resource cluster, and the kernel process is created by the kernel service started by the data processing framework in the resource cluster.

[0267] Then the processing module 82 is specifically configured to:

[0268] Establish an association relationship between a file identifier and a resource identifier;

[0269] When receiving the network address of the kernel service reported by the kernel service, establish a mapping relationship between the resource identifier and the network address;

[0270] Based on the association relationship, the mapping relationship, and the process identifier assigned to the kernel process by the kernel service, establish a communication connection with the kernel process.

[0271] In a possible embodiment, the transceiver module 81 is further configured to:

[0272] Receive a collaboration association request from a client; wherein, the collaboration association request carries: a collaboration identifier of a collaboration file for running and debugging the running process of a target file, and a resource identifier;

[0273] The processing module 82 is further configured to:

[0274] Based on the resource identifier, schedule schedulable resources to create a collaboration process, and allocate other running environments in the running environment pool except for one running environment to the collaboration process;

[0275] Based on the collaboration identifier, establish a collaboration connection with the collaboration process;

[0276] When receiving a collaboration running instruction for a collaborative file, call the collaboration process to run the collaborative file based on the collaboration connection, obtain and return a collaboration running result.

[0277] Based on the same inventive concept, an embodiment of the present application provides a running device for a development file, which can implement the functions corresponding to the foregoing running method of the development file from the perspective of a resource cluster. Please refer to Figure 8B , the device includes a transceiver module 801 and a processing module 802, wherein:

[0278] The transceiver module 801: is configured to receive a resource scheduling request from a server; wherein, the resource scheduling request is generated by the server based on a received resource association request from a client, and the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: pre-allocated schedulable resources, and a running environment pool pre-configured in the schedulable resources;

[0279] The processing module 802: is configured to create a kernel process based on the schedulable resources indicated by the resource identifier carried in the resource scheduling request, and allocate one running environment in the running environment pool indicated by the resource identifier to the kernel process;

[0280] The processing module 802 is further configured to: when the kernel process receives a process call request carrying the file content of the target file, run the target file based on the file content, obtain and return the file running result; wherein, the process call request is generated by the server after establishing a communication connection with the kernel process based on the file identifier and receiving a file running instruction for the target file, and is sent to the kernel process based on the communication connection.

[0281] In a possible embodiment, the transceiver module 801 is further configured to:

[0282] Before receiving a resource scheduling request from the server, receive a resource configuration instruction from the server; wherein, the resource configuration instruction is generated by the server based on the cluster identifier, schedulable identifier, and at least one running environment specification of the resource cluster carried in the resource creation instruction received from the client.

[0283] The processing module 802 is further configured to:

[0284] Allocate schedulable resources based on the schedulable identifier carried in the resource configuration instruction;

[0285] Configure a running environment pool in the schedulable resources based on at least one running environment specification carried in the resource configuration instruction.

[0286] In a possible embodiment, a data processing framework is deployed in the resource cluster, and the kernel process is created by a kernel service started by the data processing framework in the resource cluster; the kernel service is associated with a dependency installation location, and multiple dependency environments are installed at the dependency installation location.

[0287] Then the processing module 802 is specifically configured to:

[0288] For each of at least one running environment specification, perform the following operations respectively:

[0289] Based on one running environment specification, establish a basic environment from at least one dependency environment inherited from multiple dependency environments;

[0290] Configure multiple basic environments for the kernel service in the schedulable resources according to a preset quantity;

[0291] Add the multiple basic environments as multiple running environments to the running environment pool.

[0292] In a possible embodiment, the processing module 802 is further configured to:

[0293] Use the kernel service to receive a dependency installation request for the running environment pool; wherein, the dependency installation request is generated by the server based on a received dependency addition instruction for the running environment pool, and the dependency addition instruction carries the installation package identifier of one dependency environment.

[0294] Based on the installation package identifier carried by the kernel service in the dependency installation request, download an environment installation package for a dependency environment from a preset dependency environment source;

[0295] Based on the downloaded environment installation package, install a dependency environment at the dependency installation location associated with the kernel service.

[0296] In a possible embodiment, a data processing framework is deployed in the resource cluster, and the kernel process is created in the resource cluster using the data processing framework;

[0297] Then the processing module 802 is further configured to:

[0298] Establish a call channel for calling a function written in a second programming language used by the data processing framework using a third programming language; wherein, the third programming language is different from the second programming language;

[0299] Specifically, the processing module 802 is configured to:

[0300] Use the kernel process to instruct the data processing framework to run the target file according to the file content based on the call channel, and obtain the file operation result.

[0301] Please refer to Figure 9 , which is a computer device 900 provided by an embodiment of the present application. The computer device 900 may be, for example, Figure 1C the client 102 or the server 101 in. The current version and historical versions of the data storage program and the application software corresponding to the data storage program may be installed on the computer device 900. The computer device 900 includes a processor 980 and a memory 920. In some embodiments, the computer device 900 may include a display unit 940, and the display unit 940 includes a display panel 941 for displaying user interaction operation interfaces, etc.

[0302] In a possible embodiment, the display panel 941 may be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED), etc.

[0303] The processor 980 is used to read a computer program and then execute the method defined by the computer program. For example, the processor 980 reads a data storage program or file, etc., so as to run the data storage program on the computer device 900 and display the corresponding interface on the display unit 940. The processor 980 may include one or more general-purpose processors, and may also include one or more DSPs (Digital Signal Processors) for performing related operations to implement the technical solutions provided by the embodiments of the present application.

[0304] The memory 920 generally includes an internal memory and an external memory. The internal memory may be a random access memory (RAM), a read-only memory (ROM), a cache (CACHE), etc. The external memory may be a hard disk, an optical disc, a USB flash drive, a floppy disk, or a tape drive, etc. The memory 920 is used to store computer programs and other data. The computer programs include application programs corresponding to each client, etc. The other data may include an operating system or data generated after the application program is run. The data includes system data (such as configuration parameters of the operating system) and user data. In the embodiments of the present application, the computer program is stored in the memory 920, and the processor 980 executes the computer program in the memory 920 to implement any of the methods described in the previous figures.

[0305] As an embodiment, the memory 920 pre-stores pre-configured resources, and the pre-configured resources include: schedulable resources pre-allocated from a resource cluster, and a running environment pool pre-configured in the schedulable resources.

[0306] When the processor 980 receives a resource association request from a client, based on the resource identifier carried in the resource association request, it schedules the schedulable resources in the pre-configured resources corresponding to the resource identifier to create a kernel process, and allocates a running environment from the running environment pool pre-configured in the schedulable resources to the kernel process; the processor 980 establishes a communication connection with the kernel process based on the file identifier; when the processor 980 receives a file running instruction for a target file, it calls the kernel process to run the target file based on the communication connection, obtains and returns the file running result.

[0307] The above-mentioned display unit 940 is used to receive input digital information, character information, or contact touch operations / non-contact gestures, and generate signal inputs related to the user settings and function control of the computer device 900, etc. Specifically, in the embodiments of the present application, the display unit 940 may include a display panel 941. The display panel 941, such as a touch screen, can collect touch operations of the user on or near it (such as the user using a finger, a stylus, or any suitable object or accessory to operate on the display panel 941 or near the display panel 941), and drive the corresponding connection device according to a preset program.

[0308] In a possible embodiment, the display panel 941 may include a touch detection device and a touch controller. Among them, the touch detection device detects the touch position of the player, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 980, and can receive and execute the commands sent by the processor 980.

[0309] Among them, the display panel 941 can be implemented in multiple types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the display unit 940, in some embodiments, the computer device 900 may further include an input unit 930. The input unit 930 may include an image input device 931 and other input devices 932. Among them, the other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), trackball, mouse, joystick, etc.

[0310] As an embodiment, the display unit 940 can present the file operation result and trigger operation, etc. Thus, the processor 980 can run the development file based on the received instruction or request to implement processes such as file operation or operation debugging.

[0311] In addition to the above, the computer device 900 may further include a power supply 990 for powering other modules, an audio circuit 960, a near field communication module 970, and an RF circuit 910. The computer device 900 may further include one or more sensors 950, such as an acceleration sensor, a light sensor, a pressure sensor, etc. The audio circuit 960 specifically includes a speaker 961 and a microphone 962, etc. For example, the computer device 900 can collect the user's voice through the microphone 962 and perform corresponding operations.

[0312] As an embodiment, the number of processors 980 may be one or more. The processor 980 and the memory 920 may be coupled or relatively independent.

[0313] As an embodiment, Figure 9 the processor 980 in Figure 8A may be used to implement the functions of the transceiver module 81 and the processing module 82 in Figure 8B and may also be used to implement the functions of the transceiver module 801 and the processing module 802 in

[0314] As an embodiment, Figure 9 the processor 980 in

[0315] Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed by a computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the computer program is executed, it performs the steps including those of the above method embodiments. The aforementioned storage medium includes various media that can store program codes, such as removable storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0316] Alternatively, if the above integrated units of the present invention are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solutions of the embodiments of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. For example, it is embodied through a computer program product that is stored in a storage medium and includes a computer program for causing a computer device to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media that can store program codes, such as removable storage devices, ROM, RAM, magnetic disks, or optical discs.

[0317] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these changes and modifications.

Claims

1. A method for running a development file, characterized in that: Applied to the server, including: Receive a resource association request from a client; wherein the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a schedulable resource pre-allocated from a resource cluster, and a running environment pool pre-configured in the schedulable resource; Based on the resource identifier, scheduling the schedulable resources to create a kernel process, and allocating an operating environment in the operating environment pool to the kernel process to start the kernel process; Based on the file identifier, establishing a communication connection with the kernel process; When a file execution instruction for the target file is received, the kernel process is called based on the communication connection to execute the target file, and a file execution result is obtained and returned.

2. The method according to claim 1, characterized in that Before receiving the resource association request from the client, the method further includes: Receive a resource creation instruction from a client; wherein the resource creation instruction carries: a cluster identifier of a resource cluster, a schedulable identifier, and at least one operating environment specification; Based on the cluster identifier, the schedulable identifier and the at least one operating environment specification, a resource configuration instruction is sent to the resource cluster, so that the resource cluster allocates schedulable resources based on the schedulable identifier carried by the resource configuration instruction, configures an operating environment pool in the schedulable resources based on the at least one operating environment specification carried by the resource configuration instruction, and returns a resource configuration result; Based on the received resource configuration result, a resource identifier of the pre-configured resource is determined.

3. The method according to claim 2, characterized in that A data processing framework is deployed in the resource cluster, and sending a resource configuration instruction to the resource cluster based on the cluster identifier includes: Obtaining a data processing strategy written in a first development language; wherein the data processing strategy indicates: starting a kernel service, the kernel service being used to create and call a kernel process; Based on the task submission format provided by the data processing framework for the first development language, the data processing strategy, the schedulable identifier and at least one operating environment specification are integrated to obtain a data processing task; Based on the cluster identifier, the data processing task is submitted to the resource cluster so that the resource cluster adopts the schedulable resources allocated by the data processing framework according to the schedulable identifier, starts a kernel service, and configures an operating environment pool for the kernel service in the schedulable resources according to the at least one operating environment specification.

4. The method according to claim 3, characterized in that The determining the resource identifier of the pre-configured resource based on the received resource configuration result includes: Receive resource configuration results; wherein the resource configuration results include the network address of the kernel service; The network address is encoded to obtain a resource identifier of the preconfigured resource.

5. The method according to claim 3, characterized in that: The core service is associated with a dependency installation location, and a plurality of dependency environments are installed on the dependency installation location; Each operating environment in the operating environment pool configured for the kernel service is configured based on at least one dependent environment inherited from the multiple dependent environments.

6. The method according to claim 5, characterized in that The method further comprises: Receive a dependency addition instruction for the operating environment pool; wherein the dependency addition instruction carries an installation package identifier of a dependency environment; Based on the network address of the kernel service, a dependency installation request carrying the installation package identifier is sent to the kernel service, so that the kernel service downloads the environment installation package of the dependent environment from the dependent environment source based on the installation package identifier, and installs the dependent environment in the dependent installation location associated with the kernel service based on the environment installation package.

7. The method according to any one of claims 1 to 6, characterized in that: The step of scheduling the schedulable resources to create a kernel process based on the resource identifier, and allocating an operating environment in the operating environment pool to the kernel process includes: Based on the resource identifier, scheduling the schedulable resource to perform the following operations: Using a main process to start a kernel service; wherein the kernel service is used to create and call a kernel process; The kernel service is called and executed by a subprocess: a kernel process is created, and based on the environment states of the respective operating environments contained in the operating environment pool, an operating environment with an idle environment state is selected from the operating environment pool and allocated to the kernel process.

8. The method according to any one of claims 1 to 6, characterized in that: A data processing framework is deployed in the resource cluster, and the kernel process is created by a kernel service started by the data processing framework in the resource cluster; Then, establishing a communication connection with the kernel process based on the file identifier includes: Establishing an association relationship between the file identifier and the resource identifier; When receiving the network address of the kernel service reported by the kernel service, establishing a mapping relationship between the resource identifier and the network address; Based on the association relationship, the mapping relationship and the process identifier assigned by the kernel service to the kernel process, a communication connection with the kernel process is established.

9. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Receiving a collaboration association request from the client; wherein the collaboration association request carries: a collaboration identifier of a collaboration file for running and debugging the running process of the target file, and the resource identifier; Based on the resource identifier, scheduling the schedulable resources to create a collaboration process, and allocating other operating environments in the operating environment pool except the one operating environment to the collaboration process; Based on the collaboration identifier, establishing a collaboration connection with the collaboration process; When a collaborative running instruction for the collaborative file is received, the collaborative process is called based on the collaborative connection to run the collaborative file, and a collaborative running result is obtained and returned.

10. A method for running a development file, characterized in that: Applicable to resource clusters, including: Receive a resource scheduling request from a server; wherein the resource scheduling request is generated by the server based on a resource association request received from a client, and the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a pre-allocated schedulable resource, and a pre-configured operating environment pool in the schedulable resource; Creating a kernel process based on the schedulable resources indicated by the resource identifier carried in the resource scheduling request, and allocating an operating environment in the operating environment pool indicated by the resource identifier to the kernel process to start the kernel process; When the kernel process receives a process call request carrying the file content of the target file, the target file is run based on the file content, and the file running result is obtained and returned; wherein the process call request is generated after the server establishes a communication connection with the kernel process based on the file identifier, and when a file running instruction for the target file is received, and is sent to the kernel process based on the communication connection.

11. The method according to claim 10, characterized in that Before receiving the resource scheduling request from the server, the method further includes: Receiving a resource configuration instruction from the server; wherein the resource configuration instruction is generated by the server based on a cluster identifier, a schedulable identifier, and at least one operating environment specification of a resource cluster carried in a resource creation instruction received from a client; Allocate schedulable resources based on the schedulable identifier carried by the resource configuration instruction; Based on the at least one operating environment specification carried by the resource configuration instruction, an operating environment pool is configured in the schedulable resources.

12. The method according to claim 11, characterized in that A data processing framework is deployed in the resource cluster, and the kernel process is created by a kernel service started by the data processing framework in the resource cluster; the kernel service is associated with a dependency installation location, and a plurality of dependency environments are installed on the dependency installation location; Then, configuring an operating environment pool in the schedulable resources based on the at least one operating environment specification carried by the resource configuration instruction includes: For the at least one operating environment specification, perform the following operations respectively: Establishing a basic environment based on an operating environment specification and at least one dependent environment inherited from the multiple dependent environments; According to a preset number, configuring a plurality of the basic environments for the kernel service in the schedulable resources; Add multiple configured basic environments as multiple running environments to the running environment pool.

13. The method according to claim 12, characterized in that The method further comprises: The kernel service is used to receive a dependency installation request for the operating environment pool; wherein the dependency installation request is generated by the server based on a received dependency addition instruction for the operating environment pool, and the dependency addition instruction carries an installation package identifier of a dependency environment; Using the kernel service to download the environment installation package of the dependent environment from a preset dependent environment source based on the installation package identifier carried in the dependent installation request; Based on the downloaded environment installation package, the one dependent environment is installed in the dependent installation location associated with the kernel service.

14. The method according to any one of claims 10 to 13, characterized in that: A data processing framework is deployed in the resource cluster, and the kernel process is created in the resource cluster using the data processing framework; the method further includes: Establishing a calling channel for using a third development language to call a function written in a second programming language used by the data processing framework; wherein the third development language is different from the second programming language; Then, the step of running the target file based on the file content to obtain the file running result includes: The kernel process is used to instruct the data processing framework to run the target file according to the file content based on the calling channel to obtain a file running result.

15. A device for running a development file, characterized in that: Applied to the server, including: The transceiver module is used to receive a resource association request from a client; wherein the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a schedulable resource pre-allocated from a resource cluster, and a running environment pool pre-configured in the schedulable resource; A processing module is used for scheduling the schedulable resources to create a kernel process based on the resource identifier, and allocating an operating environment in the operating environment pool to the kernel process to start the kernel process; The processing module is also used to: establish a communication connection with the kernel process based on the file identifier; The transceiver module is also used for: upon receiving a file execution instruction for the target file, calling the kernel process based on the communication connection to execute the target file, and obtaining and returning a file execution result.

16. A device for running a development file, characterized in that: Applicable to resource clusters, including: The transceiver module is used to receive a resource scheduling request from a server; wherein the resource scheduling request is generated by the server based on a resource association request received from a client, and the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a pre-allocated schedulable resource, and a pre-configured operating environment pool in the schedulable resource; Processing module: used for creating a kernel process based on the schedulable resources indicated by the resource identifier carried in the resource scheduling request, and allocating an operating environment in the operating environment pool indicated by the resource identifier to the kernel process to start the kernel process; The processing module is also used to: when the kernel process receives a process call request carrying the file content of the target file, run the target file based on the file content, obtain and return the file running result; wherein the process call request is generated after the server establishes a communication connection with the kernel process based on the file identifier and when a file running instruction for the target file is received, and is sent to the kernel process based on the communication connection.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

18. A computer device, characterized in that: include: A memory for storing program instructions; A processor, configured to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 14 according to the obtained program instructions.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 14.

Images