Development file running method and device, equipment and storage medium

Abstract

The invention provides a running method and device of a development file, equipment and a storage medium, and is used for solving the problem of relatively low file running efficiency. The method at least comprises the following steps: receiving a resource association request from a client; wherein the resource association request carries a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resources comprise schedulable resources pre-allocated from a resource cluster and an operating environment pool pre-configured in the schedulable resources; based on the resource identifier, scheduling the schedulable resource to create a kernel process, and allocating one operating environment in the operating environment pool to the kernel process; establishing a communication connection with the kernel process based on the file identifier; and when a file running instruction for the target file is received, calling the kernel process to run the target file based on the communication connection, and obtaining and returning a file running result. The starting time consumption of the kernel process is reduced through resource presetting, so that the file running efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for operating a development file. Background Art

[0002] With the continuous development of technology, more and more clients can instruct the server to run development files to obtain file running results. For example, in the Jupyter Notebook browser, you can create and edit development files containing code, text, formulas, charts, and other contents. These files are saved in ipynb (IPython Notebook) format. Therefore, the Jupyter Notebook browser can instruct the Jupyter Notebook server to run these development files to achieve data exploration, data analysis, machine learning experiments, etc.

[0003] In the related art, taking the Jupyter Notebook server as an example, the method for running the development file is usually: When receiving a file open instruction for an ipynb file, the Jupyter Notebook server generates the operating environment specifications and operating resource requirements of the kernel process for running the ipynb file based on the ipynb file; the Jupyter Notebook server sends a resource request to the YARN cluster based on the operating environment specifications and operating resource requirements. After receiving the resource request, the YARN cluster schedules resources to create a kernel process according to the operating resource requirements, and configures the operating environment for the kernel process according to the operating environment specifications; the Jupyter Notebook server establishes a communication connection between the ipynb file and the kernel process.

[0004] When receiving the file running instruction for this ipynb file, the Jupyter Notebook server calls the kernel process to run the ipynb file based on the communication connection corresponding to the ipynb file to obtain the file running result.

[0005] However, under the related technology, each time a development file is opened, a series of resource preparation processes must be performed before a kernel process can be started or restarted, and then the development file can be run after the communication connection between the server and the kernel process is established. Since the kernel process startup or restart process takes a long time, the file operation efficiency of the development file is low and cannot meet the current efficient development needs. Summary of the invention

[0006] The embodiments of the present application provide a method, apparatus, device and storage medium for running a development file, which are used to solve the problem of low file running efficiency of the development file.

[0007] In a first aspect, a method for running a development file is provided, which is applied to a server and includes: Receive a resource association request from a client; wherein the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a schedulable resource pre-allocated from a resource cluster, and a running environment pool pre-configured in the schedulable resource; Based on the resource identifier, scheduling the schedulable resource to create a kernel process, and allocating an operating environment in the operating environment pool to the kernel process; Based on the file identifier, establishing a communication connection with the kernel process; When a file execution instruction for the target file is received, the kernel process is called based on the communication connection to execute the target file, and a file execution result is obtained and returned.

[0008] In a second aspect, a method for running a development file is provided, which is applied to a resource cluster and includes: Receive a resource scheduling request from a server; wherein the resource scheduling request is generated by the server based on a resource association request received from a client, and the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a pre-allocated schedulable resource, and a pre-configured operating environment pool in the schedulable resource; Creating a kernel process based on the schedulable resources indicated by the resource identifier carried in the resource scheduling request, and allocating an operating environment in the operating environment pool indicated by the resource identifier to the kernel process; When the kernel process receives a process call request carrying the file content of the target file, the target file is run based on the file content, and the file running result is obtained and returned; wherein the process call request is generated after the server establishes a communication connection with the kernel process based on the file identifier, and when a file running instruction for the target file is received, and is sent to the kernel process based on the communication connection.

[0009] In a third aspect, a development file operation device is provided, which is applied to a server, and includes: The transceiver module is used to receive a resource association request from a client; wherein the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a schedulable resource pre-allocated from a resource cluster, and a running environment pool pre-configured in the schedulable resource; A processing module is used for scheduling the schedulable resources to create a kernel process based on the resource identifier, and allocating an operating environment in the operating environment pool to the kernel process; The processing module is also used to: establish a communication connection with the kernel process based on the file identifier; The transceiver module is also used for: upon receiving a file execution instruction for the target file, calling the kernel process based on the communication connection to execute the target file, and obtaining and returning a file execution result.

[0010] Optionally, the transceiver module is further used for: Before receiving the resource association request from the client, receiving a resource creation instruction from the client; wherein the resource creation instruction carries: a cluster identifier of the resource cluster, a schedulable identifier and at least one operating environment specification; Based on the cluster identifier, the schedulable identifier and the at least one operating environment specification, a resource configuration instruction is sent to the resource cluster, so that the resource cluster allocates schedulable resources based on the schedulable identifier carried by the resource configuration instruction, configures an operating environment pool in the schedulable resources based on the at least one operating environment specification carried by the resource configuration instruction, and returns a resource configuration result; The processing module is further configured to determine a resource identifier of the pre-configured resource based on the received resource configuration result.

[0011] Optionally, a data processing framework is deployed in the resource cluster, and the processing module is specifically used to: Obtaining a data processing strategy written in a first development language; wherein the data processing strategy indicates: starting a kernel service, the kernel service being used to create and call a kernel process; Based on the task submission format provided by the data processing framework for the first development language, the data processing strategy, the schedulable identifier and at least one operating environment specification are integrated to obtain a data processing task; Based on the cluster identifier, the data processing task is submitted to the resource cluster so that the data processing framework schedules the schedulable resources allocated in the resource cluster according to the schedulable identifier, starts a kernel service, and configures an operating environment pool for the kernel service in the schedulable resources according to the at least one operating environment specification.

[0012] Optionally, the transceiver module is specifically used to: receive a resource configuration result; wherein the resource configuration result includes a network address of the kernel service; The processing module is specifically used to: encode the network address to obtain the resource identifier of the pre-configured resource.

[0013] Optionally, the kernel service is associated with a dependency installation location, and a plurality of dependency environments are installed on the dependency installation location; Each operating environment in the operating environment pool configured for the kernel service is configured based on at least one dependent environment inherited from the multiple dependent environments.

[0014] Optionally, the transceiver module is further used for: Receive a dependency addition instruction for the operating environment pool; wherein the dependency addition instruction carries an installation package identifier of a dependency environment; Based on the network address of the kernel service, a dependency installation request carrying the installation package identifier is sent to the kernel service, so that the kernel service downloads the environment installation package of the dependent environment from the dependent environment source based on the installation package identifier, and installs the dependent environment in the dependent installation location associated with the kernel service based on the environment installation package.

[0015] Optionally, the processing module is specifically used for: Based on the resource identifier, scheduling the schedulable resource to perform the following operations: Using a main process to start a kernel service; wherein the kernel service is used to create and call a kernel process; The kernel service is called and executed by a subprocess: a kernel process is created, and based on the environment states of the respective operating environments contained in the operating environment pool, an operating environment with an idle environment state is selected from the operating environment pool and allocated to the kernel process.

[0016] Optionally, a data processing framework is deployed in the resource cluster, and the kernel process is created by a kernel service started by the data processing framework in the resource cluster; Then the processing module is specifically used for: Establishing an association relationship between the file identifier and the resource identifier; When receiving the network address of the kernel service reported by the kernel service, establishing a mapping relationship between the resource identifier and the network address; Based on the association relationship, the mapping relationship and the process identifier assigned by the kernel service to the kernel process, a communication connection with the kernel process is established.

[0017] Optionally, the transceiver module is further used for: Receiving a collaboration association request from the client; wherein the collaboration association request carries: a collaboration identifier of a collaboration file for running and debugging the running process of the target file, and the resource identifier; The processing module is also used for: Based on the resource identifier, scheduling the schedulable resources to create a collaboration process, and allocating other operating environments in the operating environment pool except the one operating environment to the collaboration process; Based on the collaboration identifier, establishing a collaboration connection with the collaboration process; When a collaborative running instruction for the collaborative file is received, the collaborative process is called based on the collaborative connection to run the collaborative file, and a collaborative running result is obtained and returned.

[0018] In a fourth aspect, a development file operation device is provided, which is applied to a resource cluster, including: The transceiver module is used to receive a resource scheduling request from a server; wherein the resource scheduling request is generated by the server based on a resource association request received from a client, and the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a pre-allocated schedulable resource, and a pre-configured operating environment pool in the schedulable resource; Processing module: used for creating a kernel process based on the schedulable resources indicated by the resource identifier carried in the resource scheduling request, and allocating an operating environment in the operating environment pool indicated by the resource identifier to the kernel process; The processing module is also used to: when the kernel process receives a process call request carrying the file content of the target file, run the target file based on the file content, obtain and return the file running result; wherein the process call request is generated after the server establishes a communication connection with the kernel process based on the file identifier and when a file running instruction for the target file is received, and is sent to the kernel process based on the communication connection.

[0019] Optionally, the transceiver module is further used for: Before receiving the resource scheduling request from the server, receiving a resource configuration instruction from the server; wherein the resource configuration instruction is generated by the server based on the cluster identifier, the schedulable identifier and at least one operating environment specification of the resource cluster carried in the resource creation instruction received from the client; The processing module is also used for: Allocate schedulable resources based on the schedulable identifier carried by the resource configuration instruction; Based on the at least one operating environment specification carried by the resource configuration instruction, an operating environment pool is configured in the schedulable resources.

[0020] Optionally, a data processing framework is deployed in the resource cluster, and the kernel process is created by a kernel service started by the data processing framework in the resource cluster; the kernel service is associated with a dependency installation location, and a plurality of dependency environments are installed on the dependency installation location; Then the processing module is specifically used for: For the at least one operating environment specification, perform the following operations respectively: Establishing a basic environment based on an operating environment specification and at least one dependent environment inherited from the multiple dependent environments; According to a preset number, configuring a plurality of the basic environments for the kernel service in the schedulable resources; Add multiple configured basic environments as multiple running environments to the running environment pool.

[0021] Optionally, the processing module is further used for: The kernel service is used to receive a dependency installation request for the operating environment pool; wherein the dependency installation request is generated by the server based on a received dependency addition instruction for the operating environment pool, and the dependency addition instruction carries an installation package identifier of a dependency environment; Using the kernel service to download the environment installation package of the one dependent environment from a preset dependent environment source based on the installation package identifier carried in the dependent installation request; Based on the downloaded environment installation package, the one dependent environment is installed in the dependent installation location associated with the kernel service.

[0022] Optionally, a data processing framework is deployed in the resource cluster, and the kernel process is created in the resource cluster using the data processing framework; Then the processing module is also used for: Establishing a calling channel for using a third development language to call a function written in a second programming language used by the data processing framework; wherein the third development language is different from the second programming language; Then the processing module is specifically used for: The kernel process is used to instruct the data processing framework to run the target file according to the file content based on the calling channel to obtain a file running result.

[0023] According to a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method described in the first aspect or the second aspect.

[0024] A sixth aspect provides a computer device, including: A memory for storing program instructions; A processor, configured to call a program instruction stored in the memory, and execute the method according to the first aspect or the second aspect according to the obtained program instruction, including: Receive a resource association request from a client; wherein the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a schedulable resource pre-allocated from a resource cluster, and a running environment pool pre-configured in the schedulable resource; Based on the resource identifier, scheduling the schedulable resource to create a kernel process, and allocating an operating environment in the operating environment pool to the kernel process; Based on the file identifier, establishing a communication connection with the kernel process; When a file execution instruction for the target file is received, the kernel process is called based on the communication connection to execute the target file, and a file execution result is obtained and returned.

[0025] In a seventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method as described in the first aspect or the second aspect.

[0026] In an embodiment of the present application, preconfigured resources are preconfigured in the resource cluster to achieve resource preheating. When a resource association request carrying the file identifier of the target file and the resource identifier of the preconfigured resource is received, the kernel process can be started or restarted directly based on the preconfigured resource. There is no need to go through a series of resource preparation work after selecting the target file, which greatly reduces the time consumption of starting or restarting the kernel process. Then, when a file execution instruction for the target file is received, the kernel process can be efficiently called to execute the target file based on the communication connection with the kernel process, and the file execution result can be obtained and returned, effectively improving the file execution efficiency.

[0027] Furthermore, the pre-configured resources include a pool of operating environments pre-configured in the schedulable resources. After the kernel process is created, an operating environment can be allocated to the kernel process so that the kernel process can run the target file in an independent operating environment, avoiding operating interference that may cause errors in the operating results, and improving the accuracy of file operation.

[0028] Furthermore, as resident resources, when the operating environment pool in the preconfigured resources contains multiple operating environments, a kernel process can be started or restarted for each of the multiple files based on the preconfigured resources. That is, multiple kernel processes that can run independently can be started or restarted at the same time in the schedulable resources. Under the premise of ensuring the accuracy of file operation, unnecessary resource occupation caused by isolating multiple kernel processes is greatly reduced, thereby improving the file operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1A A schematic diagram of an application scenario of the method for running a development file provided in an embodiment of the present application; Figure 1B A schematic diagram of a principle of an operating method of a development file in the related art; Figure 1C A schematic diagram of an application field of the method for operating a development file provided in an embodiment of the present application; Figure 2 A flowchart of a method for operating a development file provided in an embodiment of the present application; Figure 3A A schematic diagram of a principle of a method for operating a development file provided in an embodiment of the present application; Figure 3B A schematic diagram of a method for operating a development file provided in an embodiment of the present application Figure 2 ; Figure 4A A flow chart of the method for running the development file provided in the embodiment of the present application Figure 2 ; Figure 4B A third schematic diagram of a method for operating a development file provided in an embodiment of the present application; Figure 4C A fourth schematic diagram of a principle of a method for operating a development file provided in an embodiment of the present application; Figure 5A A schematic diagram of a method for operating a development file provided in an embodiment of the present application; Figure 5B A flow chart of the method for running the development file provided in the embodiment of the present application Figure 2 ; Figure 5C A schematic diagram of a method for operating a development file provided in an embodiment of the present application Figure 6 ; Figure 6 A flowchart diagram of a method for operating a development file provided in an embodiment of the present application is shown in FIG3; Fig. 7A An interactive schematic diagram of a method for operating a development file provided in an embodiment of the present application; Figure 7B A schematic diagram of a method for operating a development file provided in an embodiment of the present application is shown in FIG7; Fig. 8A A structural schematic diagram of a device for running a development file provided in an embodiment of the present application; Figure 8B A structural diagram of a running device for a development file provided in an embodiment of the present application Figure 2 ; Fig. 9 A third structural diagram of a device for running a development file provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0031] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0032] (1) Jupyter Notebook: Jupyter Notebook is a web application that can create and share documents containing live code, equations, visualizations, and narrative text. Jupyter Notebook is essentially an interactive computing environment that is mainly used to perform tasks in many data science fields, such as data cleaning, data analysis, and machine learning.

[0033] (2) Enterprise Gateway: Enterprise Gateway is a gateway component used in an enterprise-level environment. It can be used to connect objects, data analysis tools, and back-end computing resources. Enterprise Gateway can coordinate objects, data analysis tools, back-end computing resources, and other tools with computing resources in an enterprise-level environment, so that data processing tasks can be performed efficiently.

[0034] (3) Kernel process: The kernel process is a process running in the background and is used to execute the code entered by the front end. Taking IPython Kernel as an example, it is the "execution engine" of Python code. When a Python statement (such as print("Hello, World!")) is entered in a code cell in the Jupyter Notebook browser, IPython Kernel can execute this Python statement and return the execution result to the Jupyter Notebook browser for display.

[0035] (4) Pyenv: Pyenv is a tool for managing Python versions. It can install, switch, and use multiple versions of Python in a system, and switch between various Python versions.

[0036] (5) Spark Framework and Spark Context: The Spark framework is a fast, general, and scalable distributed data processing framework, mainly used for large-scale data processing and analysis. The Spark framework provides an efficient and flexible computing model that can process various types of data, including structured data (such as table data in databases), semi-structured data (such as XML files, JSON files), and unstructured data (such as text, images, and videos).

[0037] Spark Context is a core component in the Spark framework, responsible for coordinating and managing all resources and operations related to the Spark framework.

[0038] (6)Java Gateway and JVM Gateway: Java Gateway refers to a channel that connects different Java components, systems, or interacts between Java programs and external systems in a Java language-related development environment. It mainly focuses on the connection and communication functions within the scope of Java language technology.

[0039] JVM Gateway is a connection function related to the Java Virtual Machine (JVM). JVM Gateway is a bridge for interaction between the JVM environment and other external environments (which may be programs written in other languages, different computing frameworks, or external services, etc.).

[0040] (7) Monkey Patch: Monkey Patch is a technology that dynamically modifies classes or modules at runtime, allowing the behavior of existing classes, functions, or objects to be modified or extended without modifying the original code.

[0041] It should be noted that the embodiments of the present application involve data operations such as sending and receiving requests and scheduling resources. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0042] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0043] The following is a brief introduction to the application fields of the development file operation method provided in the embodiment of the present application.

[0044] With the continuous development of technology, more and more clients can instruct the server to run development files to obtain file running results. For example, in the Jupyter Notebook browser, you can create and edit development files containing code, text, formulas, charts, and other contents. These files are saved in ipynb (IPython Notebook) format. Therefore, the Jupyter Notebook browser can instruct the Jupyter Notebook server to run these development files to achieve data exploration, data analysis, machine learning experiments, etc.

[0045] Please refer to Figure 1A , a development file named "XXXXXX / xxxxxxxx / xxxxx.ipynb" is displayed in the Jupyter Notebook browser. In the "In[1]" cell of the development file, you can enter the code "printt('Hello world!')". The Jupyter Notebook browser can instruct the Jupyter Notebook server to run the development file, obtain the file running result "Hello world!", and display the file running result.

[0046] In the related technology, taking Jupyter Notebook server as an example, please refer to Figure 1B, the development file is usually run as: When receiving a file open instruction for an ipynb file, the Jupyter Notebook server generates the operating environment specifications and operating resource requirements of the kernel process for running the ipynb file based on the ipynb file; the Jupyter Notebook server sends a resource request to the YARN cluster based on the operating environment specifications and operating resource requirements. After receiving the resource request, the YARN cluster schedules resources to create a kernel process according to the operating resource requirements, and configures the operating environment for the kernel process according to the operating environment specifications; the YARN cluster registers the kernel process in the Jupyter Notebook server to establish a communication connection between the ipynb file and the kernel process.

[0047] When receiving the file running instruction for this ipynb file, the Jupyter Notebook server calls the kernel process to run the ipynb file based on the communication connection corresponding to the ipynb file to obtain the file running result.

[0048] However, under the related technology, each time a development file is opened, a series of resource preparation processes must be performed before a kernel process can be started or restarted, and then the development file can be run after the communication connection between the server and the kernel process is established. Since the kernel process startup or restart process takes a long time, the file operation efficiency of the development file is low and cannot meet the current efficient development needs.

[0049] In order to solve the problem of low file operation efficiency of development files, the present application proposes a method for operating development files. The method is applied to the server side, in which a resource association request is received from the client. The resource association request carries: the file identifier of the target file and the resource identifier of the preconfigured resources; the preconfigured resources include: schedulable resources pre-allocated from the resource cluster, and an operating environment pool pre-configured in the schedulable resources. Based on the resource identifier, the schedulable resources are scheduled to create a kernel process, and an operating environment in the operating environment pool is allocated to the kernel process. Based on the file identifier, a communication connection with the kernel process is established. When a file operation instruction for the target file is received, the kernel process is called based on the communication connection to run the target file, and the file operation result is obtained and returned.

[0050] In an embodiment of the present application, preconfigured resources are preconfigured in the resource cluster to achieve resource preheating. When a resource association request carrying the file identifier of the target file and the resource identifier of the preconfigured resource is received, the kernel process can be started or restarted directly based on the preconfigured resource. There is no need to go through a series of resource preparation work after selecting the target file, which greatly reduces the time consumption of starting or restarting the kernel process. Then, when a file execution instruction for the target file is received, the kernel process can be efficiently called to execute the target file based on the communication connection with the kernel process, and the file execution result can be obtained and returned, effectively improving the file execution efficiency.

[0051] Furthermore, the pre-configured resources include a pool of operating environments pre-configured in the schedulable resources. After the kernel process is created, an operating environment can be allocated to the kernel process so that the kernel process can run the target file in an independent operating environment, avoiding operating interference that may cause errors in the operating results, and improving the accuracy of file operation.

[0052] Furthermore, as resident resources, when the operating environment pool in the preconfigured resources contains multiple operating environments, a kernel process can be started or restarted for each of the multiple files based on the preconfigured resources. That is, multiple kernel processes that can run independently can be started or restarted at the same time in the schedulable resources. Under the premise of ensuring the accuracy of file operation, unnecessary resource occupation caused by isolating multiple kernel processes is greatly reduced, thereby improving the file operation efficiency.

[0053] The following describes the application scenarios of the development file operation method provided in this application.

[0054] Please refer to Figure 1C , which is a schematic diagram of an application scenario of the operation method of the development file provided in this application. The application scenario includes a client 101, a server 102 and a resource cluster 103; the client 101 and the server 102 can communicate with each other, and the server 102 and the resource cluster 103 can communicate with each other; the communication method can be wired communication technology, for example, by connecting a network cable or a serial port cable; or wireless communication technology can be used for communication, for example, by Bluetooth or wireless fidelity (WIFI) and other technologies, without specific restrictions.

[0055] The client 101 generally refers to a device that can write files, send requests, or receive file execution results, etc., including, for example, a terminal device, a third-party application that can be accessed by the terminal device, or a web page that can be accessed by the terminal device, etc. The server 102 generally refers to a device that can receive requests, schedule resources to execute files, etc., including, for example, a terminal device or a server, etc. The resource cluster 103 generally refers to a device cluster that includes multiple resources, etc., including, for example, a terminal device or a server, etc.

[0056] Terminal devices include but are not limited to mobile phones, computers, smart medical devices, smart home appliances, vehicle-mounted terminals or aircraft, etc. Servers include but are not limited to cloud servers, local servers or associated third-party servers, etc. Client 101, server 102 and resource cluster 103 can all use cloud computing to reduce the occupation of local computing resources; cloud storage can also be used to reduce the occupation of local storage resources.

[0057] In response to the association operation triggered for the target file and the pre-configured resource, the client 101 sends a resource association request to the server 102 , where the resource association request is used to associate the target file with the pre-configured resource.

[0058] The server 102 receives the resource association request sent by the client 101, and the server 102 starts the kernel process for the target file based on the pre-configured resources in the resource cluster 103. In response to the run operation triggered for the target file, the client 101 sends a file run instruction to the server 102. The server 102 receives the file run instruction, and calls the kernel process started for the target file in the resource cluster 103, runs the target file, and obtains the file run result.

[0059] The server 102 sends the file operation result to the client 101 , and the client 101 receives the file operation result and presents the file operation result.

[0060] The following is based on Figure 1C , from the perspective of the server, the operation method of the development file provided by the embodiment of this application is specifically introduced. Please refer to Figure 2 , which is a flow chart of the method for operating the development file provided in an embodiment of the present application.

[0061] S201, receiving a resource association request from a client.

[0062] The client can be used to create a development file or open a created development file, and can also be used to write file content in the development file using a selected development language. A development file is a file written in a development language that can execute a function by running.

[0063] The client may also present at least one resource identifier of each preconfigured resource. The preconfigured resources include: schedulable resources pre-allocated from the resource cluster, and a running environment pool pre-configured in the schedulable resources. The schedulable resources may be resources deployed within a geographical range in a distributed resource cluster; a certain amount of resources in a resource cluster; or resources in a resource cluster that are divided into the same resource group, etc., without specific restrictions.

[0064] The resource cluster can provide a variety of resources, including computing resources, storage resources, control nodes, execution nodes, etc. The resource cluster can be a yarn cluster, a mesos cluster, a kubernetes cluster, a slurm cluster, etc., without specific limitation. In the embodiment of the present application, the yarn cluster is taken as an example for introduction.

[0065] The runtime environment pool contains at least one runtime environment, which may include at least one installed language version of a development language, such as Pyenv; it may also include dependent environments required for running files, such as front-end styles, functional functions, developed functional modules, etc., without specific restrictions.

[0066] The resource identifier is used to uniquely identify a preconfigured resource. The resource identifier may include at least one of the following information: the number and name of the preconfigured resource, the resource type of the resource cluster, the name of the schedulable resource, the name of each operating environment in the operating environment pool, etc.

[0067] The resource association request may carry the file identifier of the target file and the resource identifier of the preconfigured resource. The file identifier of the target file may uniquely represent the target file, and the file identifier may include at least one of the information such as the name, number, and storage location of the target file in the client or server. The resource association request may be used to indicate that the target file is associated with the preconfigured resource, so that a kernel process may be started based on the preconfigured resource to run the target file.

[0068] Please refer to Figure 3A , the client presents the file identifiers of multiple files, including file identifier_1, file identifier_2, ... and file identifier_N. The client also presents the resource identifiers of multiple pre-configured resources, including resource identifier_1, resource identifier_2, ... and resource identifier_M. In response to the association operation triggered for file identifier_1 and resource identifier_M, a resource association request is generated. The client sends the resource association request to the server.

[0069] Please refer to Figure 3B, the client presents the file identifiers of multiple files, including file identifier_1, file identifier_2, file identifier_3, file identifier_4, file identifier_5, file identifier_6, file identifier_7 and file identifier_8. In response to the file open operation triggered for file identifier_1, a file open instruction is generated. The client sends the file open instruction to the server, and the server obtains the template content corresponding to the file type based on the file type carried in the file open instruction. The server returns the template content to the client, and the client receives and presents the template content.

[0070] In response to a selection operation triggered by a resource association button for a target file, the client presents a pop-up window containing resource identifiers of multiple pre-configured resources. The resource identifiers of the multiple pre-configured resources include resource identifier_1, resource identifier_2, and resource identifier_3. In response to a confirmation selection operation triggered by resource identifier_3, a resource association request is generated, and the client sends the resource association request to the server.

[0071] As an embodiment, the preconfigured resources may be preconfigured according to the instructions of the client, so that the corresponding preconfigured resources can be configured for personalized scenarios; or they may be preconfigured automatically according to a preset configuration policy, so that an efficient configuration process can be achieved through a unified standard, etc., without specific limitation. By preconfiguring resources to achieve resource preheating, the preconfigured resources can be directly used during the process of starting or restarting a kernel process, without the need for a series of resource preparations, which can effectively improve the file operation efficiency when the kernel is used to run the file.

[0072] The following is a detailed description of the process of creating pre-configured resources according to the client's instructions. Please refer to Figure 4A .

[0073] S401, receiving a resource creation instruction from a client.

[0074] The resource creation instruction carries the cluster identifier of the resource cluster, the schedulable identifier and at least one operating environment specification. The cluster identifier is used to uniquely identify the resource cluster. The cluster identifier can be the name, number, etc. of the resource cluster, such as yarn is used to identify the yarn cluster. The schedulable identifier is used to indicate the resources in the resource cluster that can be divided into a preconfigured resource, such as control nodes, execution nodes, CPU performance and storage space. The schedulable identifier can be the name, number, resource quantity, etc. of the schedulable resource, and there is no specific limitation. The operating environment specification can include at least one of the version of the development language, the framework name of the data processing framework, the name of the dependent environment, etc. The operating environment specification indicates the dependent environment required to run a kernel process with a processing capability.

[0075] Please refer to Figure 4B, the client presents a preconfigured resource list page, which presents a new resource button for creating preconfigured resources, and also presents multiple preconfigured resources that have been created, and presents the running status, creation time, and executable operations of each preconfigured resource. When creating a preconfigured resource, the creation page of the preconfigured resource can be overlaid on the preconfigured resource list page. In the creation page, you can enter the resource name (such as pyspark) and resource description (a brief introduction to the preconfigured resource) of the preconfigured resource. You can also select the resource type (such as yarn) of the resource cluster to which the preconfigured resource belongs, the operating environment specification (such as spark3.3+python3.6), the schedulable identifier (such as XXXX subcluster), the number of nodes and resource quantities (such as the number of cores and storage resources) corresponding to the control nodes (such as Driver) and execution nodes (such as Executor) included in the schedulable resource quantity, etc.

[0076] S402: Send a resource configuration instruction to a resource cluster based on a cluster identifier, a schedulable identifier, and at least one operating environment specification.

[0077] After receiving the resource creation instruction, the resource cluster can be determined based on the resource identifier carried by the resource creation instruction, and the resource configuration instruction can be generated based on the schedulable identifier and at least one operating environment specification. The server sends the resource configuration instruction to the resource cluster, and the resource cluster receives the resource configuration instruction sent by the server. The resource cluster allocates schedulable resources based on the schedulable identifier carried by the resource configuration instruction, and configures the operating environment pool in the schedulable resources based on at least one operating environment specification carried by the resource configuration instruction, and returns the resource configuration result. The resource configuration result can indicate whether the configuration is successful, and can also indicate the scheduling address of the pre-configured resource. The scheduling address can be used to connect to the pre-configured resource, so that the schedulable resources in the pre-configured resource can be scheduled, and the operating environment can be used to start or restart the kernel process, etc.

[0078] As an embodiment, there may be multiple methods for sending resource configuration instructions to a resource cluster. For example, based on a network protocol fusion schedulable identifier and at least one operating environment specification, the generated message is used as a resource configuration instruction, and based on the Internet Protocol (IP) address or port address of the transceiver node in the resource cluster, the resource configuration instruction is sent to the transceiver node.

[0079] For another example, when a data processing framework is deployed in a resource cluster, resource configuration instructions may also be sent based on a task submission function provided by the data processing framework, which is described in detail below.

[0080] The data processing framework is used to provide high-performance data processing capabilities, such as the Spark framework, Hadoop framework, Flink framework, Storm framework, etc., without specific limitation. In the embodiments of the present application, the Spark framework is taken as an example for introduction.

[0081] A data processing strategy written in a first development language is obtained, the data processing strategy instructs to start a kernel service, the kernel service is used to create and call a kernel process. The first development language and the second development language used by the data processing framework can be the same or different.

[0082] The kernel service can be a computing engine that has the ability to process data based on a preconfigured resource. The kernel service can create and call a kernel process based on a preconfigured resource, or create and call multiple kernel processes based on a preconfigured resource, so that the kernel service can flexibly control the use of preconfigured resources, enable multiple kernel processes to share preconfigured resources, and improve resource utilization.

[0083] Based on the task submission format provided by the data processing framework for the first development language, the data processing strategy, the schedulable identifier and at least one operating environment specification are integrated to obtain the data processing task. There are many ways to integrate, such as arranging in a specified format, filling in a preset template, or setting in a variable in the form of a key-value pair. In the form of the task that can be recognized by the data processing framework, the data processing strategy, the schedulable identifier and at least one operating environment specification can be converted into a task form that can be recognized by the data processing framework. Therefore, with the help of the high-performance data processing capabilities of the data processing framework, a kernel service can be started to create and call the kernel process. No matter what development language the data processing strategy is written in, the data processing logic therein can be executed in the resource cluster with the help of the data processing framework.

[0084] Based on the cluster identifier, the data processing task is submitted to the resource cluster so that the resource cluster uses the data processing framework to schedule the schedulable resources allocated according to the schedulable identifier, starts a kernel service, and configures an operating environment pool for the kernel service in the schedulable resources according to at least one operating environment specification.

[0085] Please refer to Figure 4CTaking the server as Jupyter Notebook server, the resource cluster as yarn cluster, and the data processing framework as Spark framework as an example, the Jupyter Notebook server obtains the pre-stored Java program, that is, the data processing strategy, and receives the resource creation instruction sent by the client; the Jupyter Notebook server encapsulates and processes the Java program, the schedulable identifier carried by the resource creation instruction, and at least one operating environment specification based on the Spark submit task submission format provided by the Spark framework to obtain the Spark task.

[0086] Submit the Spark task to the yarn cluster indicated by the cluster identifier carried in the resource creation instruction. In the yarn cluster, the Java entrypoint functional component is used as the entry point of the Java program. According to this Java program, SparkContext is created as the entry and core control node of the Spark Application. It is responsible for establishing a connection with the cluster, resource allocation and management, task scheduling and other underlying functions. SparkContext will build functional components in the Spark framework such as DAGScheduler and TaskScheduler for subsequent task scheduling and execution.

[0087] After SparkContext is created, the kernel service can be started in the Spark Application based on the schedulable flag. After the kernel service is started, its own ip:port can be reported as the network address to the Jupyter Notebook server to establish a communication channel between the Jupyter Notebook server and the kernel service. After the kernel service is started, the kernel service can also be configured with an operating environment pool based on at least one operating environment specification to complete the creation of pre-configured resources.

[0088] S403: Determine a resource identifier of a pre-configured resource based on the received resource configuration result.

[0089] When the resource configuration result indicates that the configuration is successful, a resource identifier of the preconfigured resource is generated; when the resource configuration result indicates that the configuration fails, a reconfiguration prompt of the preconfigured resource is generated to instruct to reconfigure the preconfigured resource.

[0090] When a data processing framework is deployed in a resource cluster, the kernel process may be created by a kernel service started by the data processing framework based on a data processing strategy. Then, when the configuration is successful, the resource configuration result received may also include a network address (including an IP address or a port address, etc.) of the kernel service. The network address is connected to the preconfigured resource, and thus may be used as the scheduling address described above. When determining the resource identifier of the preconfigured resource, the network address may be encoded, and the encoded network address may be used as the resource identifier of the preconfigured resource. While uniquely representing the preconfigured resource, the mapping relationship between the resource identifier and the network address may also be recorded in the form of the encoded network address. When connecting to the preconfigured resource, the network address of the preconfigured resource may be obtained by decoding the resource identifier, thereby reducing unnecessary occupation of storage resources and helping to improve storage resource utilization. When determining the resource identifier of the preconfigured resource, the creation sequence number of the preconfigured resource may also be used as the resource identifier to simplify the process of determining the resource identifier. Then, in order to connect to the preconfigured resource, the mapping relationship between the resource identifier and the network address may be stored.

[0091] Please refer to Figure 5A , taking the server as the Jupyter Notebook server, the resource cluster as the yarn cluster, the data processing framework as the Spark framework, and the server as the Jupyter Notebook server communicating with the yarn cluster through an enterprise-level gateway (such as Enterprise Gateway) as an example, the Jupyter Notebook server can send a process start instruction to the kernel service through the network address reported by the kernel service to start kernel process A; it can also send a process restart instruction to the kernel service to restart kernel process B; it can also send a process interrupt instruction to the kernel service to suspend the running process in kernel process C; it can also send a process end instruction to the kernel service to close kernel process D.

[0092] As an embodiment, when a data processing framework is deployed in a resource cluster, the kernel service can be associated with a dependency installation location, which is used to install multiple dependency environments. The kernel service can update and delete various dependency environments installed at the dependency installation location, and can also install new dependency environments at the dependency installation location, thereby achieving unified management of the operating environment pool.

[0093] The various dependency environments installed on the dependency installation location can constitute the basic and universal operating environment of the kernel process started by the kernel service. Each operating environment in the operating environment pool configured for the kernel service is configured based on at least one dependency environment inherited from multiple dependency environments. Therefore, each operating environment in the operating environment pool shares some software packages and libraries installed on the dependency installation location. The various dependency environments installed on the dependency installation location will not be deleted or uninstalled when the kernel process is shut down, and the kernel process can continue to inherit the dependency environments when it is restarted.

[0094] For example, some commonly used data analysis libraries, such as NumPy, Pandas, and Matplotlib, can be installed in the dependency installation location. These data analysis libraries are used in many data processing and analysis processes. Therefore, the operating environment in the operating environment pool configured for the kernel service can inherit these basic dependency environments without reinstalling them.

[0095] Therefore, multiple kernel processes created by a kernel service can be based on a unified multiple dependency environment, avoiding the situation where multiple kernel processes cannot collaborate to run files due to different versions or types of installed dependency environments, thereby improving the stability of file operation.

[0096] After the operating environment is assigned to the kernel process, other dependent environments can also be installed separately in the operating environment to realize a personalized operating environment. For example, the kernel process receives the dependent installation package sent by the client, and the kernel process installs the corresponding other dependent environments in its operating environment based on the received dependent installation package. Other dependent environments installed separately in the operating environment of the kernel process will be deleted or uninstalled after the kernel process is closed, and need to be reinstalled when the kernel process is restarted, so that some infrequently used dependent environments will not always occupy the resources in the dependent installation location, which can improve resource utilization.

[0097] Please refer to Figure 5B , continuing to take the resource cluster as the yarn cluster and the data processing framework as the Spark framework as an example, the dependency environment can be inherited from the dependency installation location (such as compute Pyenv), and the operating environment pool (such as the operating environment is Pyenv) is maintained for the kernel service in the background. Then, the kernel service can retain a specified number of idle operating environments in the operating environment pool, such as 5 Pyenvs, namely Pyenv_1, Pyenv_2, Pyenv_3, Pyenv_4 and Pyenv_5. When the number of idle operating environments in the operating environment pool does not reach the specified number, a new operating environment can be created in the schedulable resources based on at least one operating environment specification. When a kernel process is destroyed, the operating environment used by the kernel process is returned to the operating environment pool.

[0098] As an embodiment, when updating various dependent environments installed on the dependent installation location, taking the addition of dependencies to the dependent installation location as an example, a dependency addition instruction for the running environment pool is received, and the dependency addition instruction can come from the client, background or other components, etc., and is not specifically limited. The dependency addition instruction carries an installation package identifier of a dependent environment. Based on the network address of the kernel service, a dependency installation request carrying the installation package identifier is sent to the kernel service, so that the kernel service downloads the environment installation package of the dependent environment from the dependent environment source based on the installation package identifier, and installs a dependent environment in the dependent installation location associated with the kernel service based on the environment installation package.

[0099] Please refer to Figure 5C , the kernel service uses the pip install command to download the environment installation package from the Python Package Index (PyPI) or other specified sources, and installs the dependent environment in the dependency installation location associated with the kernel service based on the downloaded environment installation package. If you want to install a dependent environment named numpy, you can execute the pip install numpy command to install the dependent environment.

[0100] After the various dependent environments at the dependency installation location are updated, the updated various dependent environments can be packaged. Taking the Python development environment as an example, this package may include the Python interpreter and the various installed dependent environments, as well as the respective configuration information of the various dependent environments. The package can be in the form of a compressed file or a specific format, such as the tar.gz file format, so as to be transmitted and deployed in a distributed environment. After packaging, the package can be distributed to the schedulable resources according to the data transmission method of the data processing framework in the resource cluster, such as using the JVM Gateway to call the sparkcontext functional component provided by the data processing framework. The method is written as the data transmission method in the resource cluster, and the package is distributed to each execution node in the schedulable resource (such as including execution node_1, execution node_2, ... execution node_N), so that the execution node can maintain the running environment pool by inheriting various dependent environments in the package, so that the kernel processes started by the kernel service can use the same dependent environment, realize the sharing of the dependent environment at the kernel service level, simplify the process of installing the dependent environment separately for each kernel process, and improve the efficiency and consistency of file operation.

[0101] S202: Based on the resource identifier, schedule the schedulable resources to create a kernel process, and allocate an execution environment in the execution environment pool to the kernel process.

[0102] The resource identifier carried in the resource association request can be connected to the corresponding preconfigured resource, so that the schedulable resources in the preconfigured resources can be scheduled to create a kernel process, and an operating environment can be selected from the operating environment pool configured in the schedulable resources and assigned to the kernel process to complete the startup or restart of the kernel process.

[0103] When selecting an operating environment from the operating environment pool, you can select an operating environment according to the configuration order of each operating environment; you can also select an operating environment with an idle environment state based on the environmental state of each operating environment; you can also select an operating environment with an environmental priority greater than a preset priority according to the environmental priority of each operating environment. The longer the operating environment is in the idle state, the higher the environmental priority; the more dependent environments the operating environment contains, the higher the environmental priority, etc. There is no specific restriction.

[0104] As an embodiment, when creating a kernel process and allocating an operating environment, the following operations may be performed to schedule schedulable resources based on resource identifiers: A kernel service is started by a main process, and the kernel service is used to create and call the kernel process. The kernel service is called and executed by a child process (such as subprocess): a kernel process is created, and based on the environment status of each operating environment contained in the operating environment pool, an operating environment with an idle environment status is selected from the operating environment pool and assigned to the kernel process.

[0105] Since the child process has independent memory space and running space, it is isolated from the main process and other child processes. Therefore, when the kernel process is started or restarted in the child process, it will not be disturbed by other kernel processes or kernel services running in the main process. The running status, variables and resource allocation within the kernel process are independent of other kernel processes, avoiding abnormal situations such as running conflicts and errors caused by shared dependency environments and shared schedulable resources, and improving the stability of file operation.

[0106] Furthermore, when a child process is created, it can be given specific permissions and security contexts to limit its access to specific resources and sensitive data in the resource cluster, thereby effectively preventing the code running in the kernel process from obtaining specific resources and sensitive data in the resource cluster due to vulnerabilities, etc., thereby ensuring data security during file runtime.

[0107] S203: Establish a communication connection with the kernel process based on the file identifier.

[0108] Based on the file identifier, a communication connection between the server and the kernel process can be established for the target file, so that the kernel process can be instructed to run the target file through the communication connection. The communication connection is used to send data to the kernel process and receive data from the kernel process. For example, the communication connection includes the scheduling address of the preconfigured resource introduced above, and the process identifier of the kernel process; for another example, the communication connection includes the network address of the kernel service introduced above (such as an IP address or a port address, etc.), and the process identifier of the kernel process, etc., without specific limitation.

[0109] As an embodiment, when a data processing framework is deployed in a resource cluster and a kernel process is created by a kernel service started by the data processing framework in the resource cluster, the target file can be associated with the preconfigured resource by establishing an association relationship between the file identifier and the resource identifier; further, when receiving the network address of the kernel service reported by the kernel service, a mapping relationship between the resource identifier and the network address can be established to register the kernel service on the server side; a mapping relationship between the network address and the process identifier assigned by the kernel service to the kernel process can also be further established to register the kernel process on the server side, etc. Based on the communication connection established with the kernel process by the association relationship, the mapping relationship and the process identifier assigned by the kernel service to the kernel process, the target file can be connected to the kernel process, so that the file content in the target file can be directly sent to the kernel process for execution, thereby improving the file operation efficiency.

[0110] An enterprise-level gateway (such as Enterprise Gateway) may also be set up between the server and the resource cluster, so that a communication connection can be established between the enterprise-level gateway and the kernel process, which helps to achieve efficient file operation.

[0111] S204, when receiving a file execution instruction for a target file, calling a kernel process to execute the target file based on the communication connection, and obtaining and returning a file execution result.

[0112] After the target file is associated with the preconfigured resources and the kernel process is started during the association process, if a file execution instruction for the target file is received, the kernel process can be called to execute the target file based on the communication connection with the kernel process to obtain and return the file execution result.

[0113] The file execution instruction may be generated in response to a file execution operation triggered on a client, or may be generated when the file content of a target file is detected periodically and it is determined that the file content meets the execution format, etc., without any specific limitation. The execution format indicates that there are no syntax errors or uncallable function call relationships in the file content.

[0114] As an embodiment, when a preconfigured resource is associated with multiple files, multiple kernel processes are started in the preconfigured resource accordingly. Therefore, when multiple kernel processes use schedulable resources to run their respective received files, there may be abnormal situations such as resource scheduling conflicts or resource competition. Therefore, a third development language is used to pre-establish a calling channel for a function function written in a second programming language, so that a function call function written in a third development language corresponding to a function function written in the second programming language can be created at the level of the third development language, and each function call function can be used as a calling channel for calling a function written in the second programming language using the third development language. The third development language can be the same as or different from the first development language; the third development language is different from the second programming language.

[0115] At the level of the third development language, each function call function can be uniformly managed, configured and restricted, so that when calling the function written in the second programming language through each function call function, there will be no abnormal situations such as resource scheduling conflicts or resource competition, thereby achieving the effect of sharing the function functions written in the second programming language in multiple kernel processes, effectively ensuring the stability of file operation.

[0116] When a data processing framework using a second development language is deployed in a resource cluster, a kernel process is created in the resource cluster using the data processing framework. If a calling channel for using a third development language to call a function written in the second programming language used by the data processing framework is provided in the resource cluster, then after sending the file content contained in the target file to the kernel process based on the communication connection, the kernel process uses the calling channel to instruct the data processing framework to run the target file according to the file content to obtain the file running result.

[0117] For example, when the data processing framework is the Spark framework, the second development language used by the Spark framework is Python, and the Spark framework can divide pre-configured resources in the resource cluster with the Spark application. Since the Spark application can only initialize the Spark Context once, etc., and is responsible for the resource application, resource allocation and resource management of the schedulable resources in the entire Spark application, when multiple kernel processes are started based on one pre-configured resource, the Spark Context cannot be shared and used between the multiple kernel processes.

[0118] The third development language can be Java. Creating Spark Context at the Java level can better manage and allocate pre-configured resources in a unified manner. Spark Context is created at the Java level through Java Gateway to obtain the calling function, which calls Spark Context for data processing. Various parameters of Spark Context, such as application name, operation mode, resource allocation, etc., can be set centrally in Java code without having to configure them separately in different places, thereby improving the consistency and maintainability of configuration.

[0119] After the Spark Context is created at the Java level, that is, after the calling function is obtained, the kernel process can access the calling function through the JVM Gateway to achieve the purpose of calling the Spark Context. In this way, the kernel process can obtain the resource information of the Spark cluster, submit tasks, obtain calculation results, etc.

[0120] As an example, the Spark Context created at the Java level can be modified and packaged through Monkey Patch so that the way to use the Spark Context in Python is consistent with the way to use it in the original Java or Scala environment. Even if the underlying layer uses Java Gateway and JVM Gateway to interact with the Spark cluster, you can use Spark in the Python environment just like in a pure Python environment, which greatly reduces the threshold for using the call channel and improves development efficiency.

[0121] For example, in Python, you can directly call the SparkContext created at the Java level according to the calling methods of SparkContext, such as calling the method to create a distributed dataset (RDD) and calling the method to execute a Structured Query Language (SQL) query. These methods can be executed at the bottom layer by forwarding them to the SparkContext at the Java level through calling channels such as Java Gateway and JVM Gateway.

[0122] As an embodiment, in a scenario where multiple files run collaboratively, multiple kernel processes can be started in a preconfigured resource, and each kernel process runs one file. Thus, multiple files can run based on one preconfigured resource, so that multiple files can run collaboratively without generating resource interference or resource competition, and without causing abnormal situations such as code confusion.

[0123] Taking the debugging scenario as an example, a collaborative association request is received from the client, and the collaborative association request carries the collaborative identifier of the collaborative file for running and debugging the running process of the target file, as well as the resource identifier of the preconfigured resource. Based on the resource identifier, the schedulable resources are scheduled to create a kernel process as a collaborative process, and other operating environments in the operating environment pool except the operating environment for running the target file are allocated to the collaborative process. Based on the collaborative identifier, a collaborative connection is established with the collaborative process. When a collaborative running instruction for the collaborative file is received, the collaborative process is called based on the collaborative connection to run the collaborative file, and the collaborative running result is obtained and returned.

[0124] The target file and the collaborative file can each run in an independent running environment. There is no need to associate the target file and the collaborative file in the same kernel process, which avoids code context confusion and improves file running stability.

[0125] Based on the same inventive concept, from the perspective of resource cluster, the operation method of the development file provided in the embodiment of the present application is specifically introduced. Figure 6 , which is a flow chart of the method for operating the development file provided in an embodiment of the present application.

[0126] S601, receiving a resource scheduling request from a server.

[0127] The resource scheduling request is generated by the server based on the resource association request received from the client. The resource association request carries the file identifier of the target file and the resource identifier of the pre-configured resource. The pre-configured resources include pre-allocated schedulable resources and the pre-configured operating environment pool in the schedulable resources. For details, please refer to the previous introduction and will not be repeated here.

[0128] As an embodiment, before receiving a resource scheduling request from a server, a resource configuration instruction from a server is received. The resource configuration instruction is generated by the server based on a cluster identifier of a resource cluster carried by a resource creation instruction received from a client, a schedulable identifier, and at least one operating environment specification. Scheduling resources are allocated based on the schedulable identifier carried by the resource configuration instruction. Based on at least one operating environment specification carried by the resource configuration instruction, an operating environment pool is configured in the schedulable resources. For details, please refer to the previous description, which will not be repeated here.

[0129] As an embodiment, a data processing framework is deployed in a resource cluster, and a kernel process is created by a kernel service started by the data processing framework in the resource cluster. The kernel service is associated with a dependency installation location, and multiple dependency environments are installed on the dependency installation location. For details, please refer to the previous introduction, which will not be repeated here. For at least one operating environment specification, perform the following operations respectively: Based on an operating environment specification, at least one dependent environment inherited from multiple dependent environments is used to establish a basic environment. Multiple basic environments are configured for kernel services in schedulable resources according to the preset number. Multiple basic environments are added to the operating environment pool as multiple operating environments. Multiple basic environments contain the same dependent environment, and the operating environment pool is maintained in the background. The number of idle operating environments in the operating environment pool can be maintained at the preset number to avoid the situation where the operating environment pool is fully occupied, thereby improving the stability of file operation.

[0130] As an embodiment, a kernel service is used to receive a dependency installation request for an operating environment pool. The dependency installation request is generated by the server based on a received dependency addition instruction for the operating environment pool, and the dependency addition instruction carries an installation package identifier of a dependency environment. The kernel service is used to download an environment installation package of a dependency environment from a preset dependency environment source based on the installation package identifier carried in the dependency installation request. Based on the downloaded environment installation package, a dependency environment is installed in a dependency installation location associated with the kernel service. For details, please refer to the previous introduction and will not be repeated here.

[0131] S602, creating a kernel process based on the schedulable resources indicated by the resource identifier carried in the resource scheduling request, and allocating an operating environment in the operating environment pool indicated by the resource identifier to the kernel process. For details, please refer to the above introduction and will not be repeated here.

[0132] S603, when the kernel process receives a process call request carrying the file content of the target file, it runs the target file based on the file content, obtains and returns the file running result.

[0133] The process call request is generated after the server establishes a communication connection with the kernel process based on the file identifier and receives a file execution instruction for the target file, and is sent to the kernel process based on the communication connection. For details, please refer to the previous introduction and will not be repeated here.

[0134] As an embodiment, a data processing framework is deployed in a resource cluster, and a kernel process is created in the resource cluster using the data processing framework. A call channel is established for using a third development language to call a function written in a second programming language used by the data processing framework. The third development language is different from the second programming language. The kernel process is used to instruct the data processing framework to run a target file according to the file content based on the call channel to obtain a file running result. For details, please refer to the previous introduction, which will not be repeated here.

[0135] The following example uses a Jupyter Notebook server as the server, a yarn cluster as the resource cluster, a Spark framework as the data processing framework, and the Jupyter Notebook server and the yarn cluster communicate through an enterprise-level gateway (DataLab EnterpriseGateway). The client and the yarn cluster also communicate through the enterprise-level gateway (DEG) to introduce the operation method of the development file provided in the embodiment of the present application. Please refer to Fig. 7A and Figure 7B .

[0136] S701, generating a resource creation instruction in response to a resource creation operation triggered by a client.

[0137] The resource creation operation includes the operation of inputting the cluster identifier of the resource cluster, the operation of inputting the schedulable identifier, and the operation of inputting at least one operating environment specification, etc. Therefore, the resource creation instruction can carry the cluster identifier of the yarn cluster, the schedulable identifier of a resource pool, and the operating environment specification of Spark 3.3 version combined with Python 3.6, etc.

[0138] The client sends a resource creation instruction to the Jupyter Notebook server, and the Jupyter Notebook server receives the resource creation instruction through the enterprise gateway (DEG).

[0139] S702, the enterprise-level gateway (DEG) obtains a Java program that records a data processing strategy, and integrates the data processing strategy, a schedulable identifier, and at least one operating environment specification in a task submission format of the Spark framework to obtain a Spark task and submit the Spark task to the yarn cluster.

[0140] S703, in the yarn cluster, based on the data processing strategy indicated by the Spark task, in the schedulable resources indicated by the Spark task, the Java Gateway is used to create the Spark Context at the Java level, and the kernel service (Kernel Operator) is launched based on the Spark Context.

[0141] S704, the yarn cluster sends the network address of the kernel service to the enterprise gateway (DEG), the Jupyter Notebook server receives the network address of the kernel service sent by the yarn cluster through the enterprise gateway (DEG), and establishes a mapping relationship between a resource identifier and the network address.

[0142] S705, in response to a resource association operation triggered by a resource identifier of a preconfigured resource and a target file, a resource association request is generated. For example, after the client responds to a file opening operation on the target file, the file content of the target file is presented on the client; then, in the file content, in response to a resource association operation triggered by a resource identifier of a preconfigured resource, a resource association request is generated.

[0143] The client sends a resource association request to the Jupyter Notebook server, and the Jupyter Notebook server receives the resource association request through the enterprise gateway (DEG).

[0144] S706, the Jupyter Notebook server determines a network address corresponding to a resource identifier based on a mapping relationship through the enterprise gateway (DEG), and sends a process start instruction to the kernel service running in the yarn cluster based on the network address.

[0145] S707, the kernel service creates a kernel process from the schedulable resources included in the pre-configured resources, and allocates an operating environment A to the kernel process from the associated operating environment pool, thereby completing the startup of the kernel process A.

[0146] S708, the kernel process obtains the SparkContext, Spark Session and other functional functions created at the Java level using the Java Gateway through the JVM Gateway, and completes the establishment of the communication connection between the Jupyter Notebook server and the kernel process.

[0147] S709, when receiving the file execution instruction A for the target file, calling the kernel process to execute the target file based on the communication connection, and obtaining and returning the file execution result.

[0148] S709A: The client generates a file running instruction A in response to the file running operation triggered for the target file; the client sends the file running instruction A to the Jupyter Notebook server through the enterprise gateway (DEG).

[0149] S709B: The Jupyter Notebook server sends the target file to the kernel process A started by the YARN cluster through the communication connection to instruct the YARN cluster to call the kernel process A to run the target file.

[0150] S709C: After receiving the target file through the communication connection, the yarn cluster calls kernel process A to run the target file and obtains the file running result.

[0151] S709D: The YARN cluster returns the file running result to the Jupyter Notebook server through the communication connection, and the Jupyter Notebook server receives the file running result sent by the YARN cluster through the communication connection.

[0152] S709E: The Jupyter Notebook server returns the received file running result to the client, and the client receives the file running result sent by the Jupyter Notebook server.

[0153] For collaborative files, the above process is repeated, and the kernel service creates a kernel process in the schedulable resources included in the preconfigured resources, and allocates an operating environment B to the kernel process from the associated operating environment pool, completing the startup of kernel process B. When receiving the file running instruction B for the collaborative file, the kernel process is called based on the communication connection to run the target file, and the file running result is obtained and returned.

[0154] In an embodiment of the present application, the startup time of the kernel process is shortened by pre-configuring resources to be resident. Compared with the effect of starting the kernel process of at least 1 minute obtained by the traditional method of running development files, the embodiment of the present application can complete the startup of the kernel process in 2 seconds, greatly shortening the startup process of the kernel process.

[0155] Furthermore, starting a spark application with a pre-configured resource can start multiple kernel processes in the same spark application, avoiding the situation where a kernel process monopolizes the spark application and causing resource waste. Furthermore, unified dependency package management is performed with a pre-configured resource as a unit, so that the newly added dependency environment can take effect in distributed computing, and the restart of the kernel process will not cause the loss of the dependency environment at the spark application level, etc., which improves the stability of file operation.

[0156] Based on the same inventive concept, the embodiment of the present application provides a development file operation device, which can realize the functions corresponding to the aforementioned development file operation method from the perspective of the server. Fig. 8A, the device comprises a transceiver module 81 and a processing module 82, wherein: The transceiver module 81 is used to receive a resource association request from a client; wherein the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a schedulable resource pre-allocated from a resource cluster, and a running environment pool pre-configured in the schedulable resource; Processing module 82: used for scheduling schedulable resources to create a kernel process based on the resource identifier, and allocating an operating environment in the operating environment pool to the kernel process; The processing module 82 is also used to: establish a communication connection with the kernel process based on the file identifier; The transceiver module 81 is also used for: upon receiving a file execution instruction for a target file, calling a kernel process to execute the target file based on the communication connection, and obtaining and returning a file execution result.

[0157] In a possible embodiment, the transceiver module 81 is further used for: Before receiving the resource association request from the client, receiving a resource creation instruction from the client; wherein the resource creation instruction carries: a cluster identifier of the resource cluster, a schedulable identifier and at least one operating environment specification; Based on the cluster identifier, the schedulable identifier and at least one operating environment specification, a resource configuration instruction is sent to the resource cluster, so that the resource cluster allocates the schedulable resources based on the schedulable identifier carried in the resource configuration instruction, configures the operating environment pool in the schedulable resources based on the at least one operating environment specification carried in the resource configuration instruction, and returns the resource configuration result; The processing module 82 is further configured to determine a resource identifier of a pre-configured resource based on the received resource configuration result.

[0158] In a possible embodiment, a data processing framework is deployed in the resource cluster, and the processing module 82 is specifically used for: Obtaining a data processing strategy written in a first development language; wherein the data processing strategy indicates: starting a kernel service, the kernel service is used to create and call a kernel process; Based on a task submission format provided by the data processing framework for the first development language, the data processing strategy, the schedulable identifier, and at least one operating environment specification are integrated to obtain a data processing task; Based on the cluster identifier, the data processing task is submitted to the resource cluster so that the data processing framework schedules the schedulable resources allocated in the resource cluster according to the schedulable identifier, starts a kernel service, and configures an operating environment pool for the kernel service in the schedulable resources according to at least one operating environment specification.

[0159] In a possible embodiment, the transceiver module 81 is specifically used to: receive a resource configuration result; wherein the resource configuration result includes a network address of a kernel service; The processing module 82 is specifically used to encode the network address to obtain the resource identifier of the pre-configured resource.

[0160] In a possible embodiment, the kernel service is associated with a dependency installation location, and a plurality of dependency environments are installed on the dependency installation location; Each operating environment in the operating environment pool configured for the kernel service is configured based on at least one dependent environment inherited from the multiple dependent environments.

[0161] In a possible embodiment, the transceiver module 81 is further used for: Receive a dependency addition instruction for the running environment pool; wherein the dependency addition instruction carries an installation package identifier of a dependency environment; Based on the network address of the kernel service, a dependency installation request carrying an installation package identifier is sent to the kernel service, so that the kernel service downloads an environment installation package of a dependency environment from a dependency environment source based on the installation package identifier, and installs a dependency environment in a dependency installation location associated with the kernel service based on the environment installation package.

[0162] In a possible embodiment, the processing module 82 is specifically used for: Based on the resource identifier, the scheduler can perform the following operations on the schedulable resource: A kernel service is started by using a main process; wherein the kernel service is used to create and call a kernel process; Call the kernel service and use the child process to execute: create a kernel process, and based on the environment status of each operating environment contained in the operating environment pool, select an operating environment with an idle environment status from the operating environment pool and assign it to the kernel process.

[0163] In a possible embodiment, a data processing framework is deployed in the resource cluster, and the kernel process is created by a kernel service started by the data processing framework in the resource cluster; The processing module 82 is specifically used for: Establishing the association relationship between the file identifier and the resource identifier; When receiving the network address of the kernel service reported by the kernel service, a mapping relationship between the resource identifier and the network address is established; Based on the association relationship, the mapping relationship, and the process identifier assigned by the kernel service to the kernel process, a communication connection with the kernel process is established.

[0164] In a possible embodiment, the transceiver module 81 is further used for: Receive a collaboration association request from a client; wherein the collaboration association request carries: a collaboration identifier of a collaboration file for running and debugging a running process of a target file, and a resource identifier; The processing module 82 is also used for: Based on the resource identifier, the schedulable resources are scheduled to create a cooperative process, and the other operating environments except one operating environment in the operating environment pool are allocated to the cooperative process; Based on the collaboration identifier, a collaboration connection is established with the collaboration process; When a collaborative running instruction for a collaborative file is received, a collaborative process is called based on a collaborative connection to run the collaborative file, and a collaborative running result is obtained and returned.

[0165] Based on the same inventive concept, the embodiment of the present application provides a development file operation device, which can realize the functions corresponding to the aforementioned development file operation method from the perspective of resource cluster. Figure 8B , the device includes a transceiver module 801 and a processing module 802, wherein: The transceiver module 801 is used to receive a resource scheduling request from a server; wherein the resource scheduling request is generated by the server based on a resource association request received from a client, and the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a pre-allocated schedulable resource, and a pre-configured operating environment pool in the schedulable resource; Processing module 802: used to create a kernel process based on the schedulable resource indicated by the resource identifier carried in the resource scheduling request, and allocate an operating environment in the operating environment pool indicated by the resource identifier to the kernel process; The processing module 802 is also used for: when the kernel process receives a process call request carrying the file content of the target file, running the target file based on the file content, obtaining and returning the file running result; wherein, the process call request is generated after the server establishes a communication connection with the kernel process based on the file identifier, and when a file running instruction for the target file is received, and is sent to the kernel process based on the communication connection.

[0166] In a possible embodiment, the transceiver module 801 is further used for: Before receiving a resource scheduling request from a server, receiving a resource configuration instruction from the server; wherein the resource configuration instruction is generated by the server based on a cluster identifier, a schedulable identifier, and at least one operating environment specification of a resource cluster carried in a resource creation instruction received from a client; The processing module 802 is further configured to: Allocate schedulable resources based on the schedulable identifier carried by the resource configuration instruction; Based on at least one operating environment specification carried by the resource configuration instruction, an operating environment pool is configured in the schedulable resources.

[0167] In a possible embodiment, a data processing framework is deployed in the resource cluster, and the kernel process is created by a kernel service started by the data processing framework in the resource cluster; the kernel service is associated with a dependency installation location, and multiple dependency environments are installed in the dependency installation location; The processing module 802 is specifically used for: For at least one operating environment specification, perform the following operations: Based on an operating environment specification, a basic environment is established by inheriting at least one dependent environment from multiple dependent environments; Configure multiple basic environments for kernel services in schedulable resources according to the preset quantity; Add multiple basic environments as multiple running environments to the running environment pool.

[0168] In a possible embodiment, the processing module 802 is further configured to: A kernel service is used to receive a dependency installation request for the operating environment pool; wherein the dependency installation request is generated by the server based on a received dependency addition instruction for the operating environment pool, and the dependency addition instruction carries an installation package identifier of a dependency environment; Using the kernel service based on the installation package identifier carried in the dependency installation request, downloading an environment installation package of a dependency environment from a preset dependency environment source; Based on the downloaded environment installation package, install a dependent environment in the dependency installation location associated with the kernel service.

[0169] In a possible embodiment, a data processing framework is deployed in the resource cluster, and the kernel process is created in the resource cluster using the data processing framework; Then the processing module 802 is further used for: Establishing a calling channel for using a third development language to call a function written in a second programming language used by the data processing framework; wherein the third development language is different from the second programming language; The processing module 802 is specifically used for: The kernel process is used to instruct the data processing framework to run the target file according to the file content based on the calling channel to obtain the file running result.

[0170] Please refer to Fig. 9 , is a computer device 900 provided in an embodiment of the present application, and the computer device 900 can be, for example, Figure 1CThe client 102 or the server 101 in the data storage program. The current version and the historical version of the data storage program and the application software corresponding to the data storage program can be installed on the computer device 900, and the computer device 900 includes a processor 980 and a memory 920. In some embodiments, the computer device 900 may include a display unit 940, and the display unit 940 includes a display panel 941 for displaying a user interactive operation interface, etc.

[0171] In a possible embodiment, the display panel 941 may be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED).

[0172] The processor 980 is used to read the computer program and then execute the method defined by the computer program. For example, the processor 980 reads the data storage program or file, so as to run the data storage program on the computer device 900 and display the corresponding interface on the display unit 940. The processor 980 may include one or more general-purpose processors and may also include one or more DSPs (Digital Signal Processors) to perform related operations to implement the technical solutions provided in the embodiments of the present application.

[0173] The memory 920 generally includes internal memory and external memory. The internal memory may be a random access memory (RAM), a read-only memory (ROM), and a cache (CACHE), etc. The external memory may be a hard disk, an optical disk, a USB disk, a floppy disk, or a tape drive, etc. The memory 920 is used to store computer programs and other data. The computer program includes applications corresponding to each client, etc. Other data may include data generated after the operating system or application is run, and the data includes system data (such as configuration parameters of the operating system) and user data. In the embodiment of the present application, the computer program is stored in the memory 920, and the processor 980 executes the computer program in the memory 920 to implement any of the methods discussed in the previous figure.

[0174] As an embodiment, the memory 920 pre-stores pre-configured resources, and the pre-configured resources include: schedulable resources pre-allocated from a resource cluster, and an operating environment pool pre-configured in the schedulable resources.

[0175] When processor 980 receives a resource association request from a client, it schedules the schedulable resources in the preconfigured resources corresponding to the resource identifier to create a kernel process based on the resource identifier carried in the resource association request, and allocates an operating environment in the operating environment pool pre-configured in the schedulable resources to the kernel process; processor 980 establishes a communication connection with the kernel process based on the file identifier; when processor 980 receives a file execution instruction for a target file, it calls the kernel process to execute the target file based on the communication connection, and obtains and returns the file execution result.

[0176] The display unit 940 is used to receive input digital information, character information or contact touch operation / contactless gesture, and generate signal input related to user settings and function control of the computer device 900. Specifically, in the embodiment of the present application, the display unit 940 may include a display panel 941. The display panel 941 is, for example, a touch screen, which can collect user touch operations on or near it (such as operations performed by the user using fingers, stylus, or any other suitable object or accessory on or on the display panel 941), and drive corresponding connection devices according to a pre-set program.

[0177] In a possible embodiment, the display panel 941 may include two parts: a touch detection device and a touch controller. The touch detection device detects the touch position of the player, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 980, and can receive and execute the command sent by the processor 980.

[0178] The display panel 941 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the display unit 940, in some embodiments, the computer device 900 may further include an input unit 930, which may include an image input device 931 and other input devices 932, wherein the other input devices may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0179] As an embodiment, the display unit 940 can present file running results and trigger operations, etc., so that the processor 980 can run the development file based on the received instructions or requests to implement file running or running debugging and other processes.

[0180] In addition to the above, the computer device 900 may also include a power supply 990 for supplying power to other modules, an audio circuit 960, a near field communication module 970, and an RF circuit 910. The computer device 900 may also include one or more sensors 950, such as an acceleration sensor, a light sensor, a pressure sensor, etc. The audio circuit 960 specifically includes a speaker 961 and a microphone 962, etc. For example, the computer device 900 can collect the user's voice through the microphone 962 to perform corresponding operations, etc.

[0181] As an embodiment, the number of processors 980 may be one or more, and the processor 980 and the memory 920 may be coupled or relatively independently configured.

[0182] As an example, Fig. 9 The processor 980 in the embodiment can be used to implement the following Fig. 8A The functions of the transceiver module 81 and the processing module 82 in the embodiment can also be used to implement the following Figure 8B The functions of the transceiver module 801 and the processing module 802 in.

[0183] As an example, Fig. 9 The processor 980 in can be used to implement the corresponding functions of the server or terminal device discussed above.

[0184] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by a computer program, and the aforementioned computer program can be stored in a computer-readable storage medium. When the computer program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0185] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product, for example, through a computer program product, which is stored in a storage medium, including a computer program for a computer device to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0186] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for running a development file, characterized in that: Applied to the server, including: Receive a resource association request from a client; wherein the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a schedulable resource pre-allocated from a resource cluster, and a running environment pool pre-configured in the schedulable resource; Based on the resource identifier, scheduling the schedulable resource to create a kernel process, and allocating an operating environment in the operating environment pool to the kernel process; Based on the file identifier, establishing a communication connection with the kernel process; When a file execution instruction for the target file is received, the kernel process is called based on the communication connection to execute the target file, and a file execution result is obtained and returned.

2. The method according to claim 1, characterized in that: Before receiving the resource association request from the client, the method further includes: Receive a resource creation instruction from a client; wherein the resource creation instruction carries: a cluster identifier of a resource cluster, a schedulable identifier, and at least one operating environment specification; Based on the cluster identifier, the schedulable identifier and the at least one operating environment specification, a resource configuration instruction is sent to the resource cluster, so that the resource cluster allocates schedulable resources based on the schedulable identifier carried by the resource configuration instruction, configures an operating environment pool in the schedulable resources based on the at least one operating environment specification carried by the resource configuration instruction, and returns a resource configuration result; Based on the received resource configuration result, a resource identifier of the pre-configured resource is determined.

3. The method according to claim 2, characterized in that A data processing framework is deployed in the resource cluster, and sending a resource configuration instruction to the resource cluster based on the cluster identifier includes: Obtaining a data processing strategy written in a first development language; wherein the data processing strategy indicates: starting a kernel service, the kernel service being used to create and call a kernel process; Based on the task submission format provided by the data processing framework for the first development language, the data processing strategy, the schedulable identifier and at least one operating environment specification are integrated to obtain a data processing task; Based on the cluster identifier, the data processing task is submitted to the resource cluster so that the resource cluster adopts the schedulable resources allocated by the data processing framework according to the schedulable identifier, starts a kernel service, and configures an operating environment pool for the kernel service in the schedulable resources according to the at least one operating environment specification.

4. The method according to claim 3, characterized in that: The determining the resource identifier of the pre-configured resource based on the received resource configuration result includes: Receive resource configuration results; wherein the resource configuration results include the network address of the kernel service; The network address is encoded to obtain a resource identifier of the preconfigured resource.

5. The method according to claim 3, characterized in that: The core service is associated with a dependency installation location, and a plurality of dependency environments are installed on the dependency installation location; Each operating environment in the operating environment pool configured for the kernel service is configured based on at least one dependent environment inherited from the multiple dependent environments.

6. The method according to claim 5, characterized in that The method further comprises: Receive a dependency addition instruction for the operating environment pool; wherein the dependency addition instruction carries an installation package identifier of a dependency environment; Based on the network address of the kernel service, a dependency installation request carrying the installation package identifier is sent to the kernel service, so that the kernel service downloads the environment installation package of the dependent environment from the dependent environment source based on the installation package identifier, and installs the dependent environment in the dependent installation location associated with the kernel service based on the environment installation package.

7. The method according to any one of claims 1 to 6, characterized in that: The step of scheduling the schedulable resources to create a kernel process based on the resource identifier, and allocating an operating environment in the operating environment pool to the kernel process includes: Based on the resource identifier, scheduling the schedulable resource to perform the following operations: Using a main process to start a kernel service; wherein the kernel service is used to create and call a kernel process; The kernel service is called and executed by a subprocess: a kernel process is created, and based on the environment states of the respective operating environments contained in the operating environment pool, an operating environment with an idle environment state is selected from the operating environment pool and allocated to the kernel process.

8. The method according to any one of claims 1 to 6, characterized in that: A data processing framework is deployed in the resource cluster, and the kernel process is created by a kernel service started by the data processing framework in the resource cluster; Then, establishing a communication connection with the kernel process based on the file identifier includes: Establishing an association relationship between the file identifier and the resource identifier; When receiving the network address of the kernel service reported by the kernel service, establishing a mapping relationship between the resource identifier and the network address; Based on the association relationship, the mapping relationship and the process identifier assigned by the kernel service to the kernel process, a communication connection with the kernel process is established.

9. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Receiving a collaboration association request from the client; wherein the collaboration association request carries: a collaboration identifier of a collaboration file for running and debugging the running process of the target file, and the resource identifier; Based on the resource identifier, scheduling the schedulable resources to create a collaboration process, and allocating other operating environments in the operating environment pool except the one operating environment to the collaboration process; Based on the collaboration identifier, establishing a collaboration connection with the collaboration process; When a collaborative running instruction for the collaborative file is received, the collaborative process is called based on the collaborative connection to run the collaborative file, and a collaborative running result is obtained and returned.

10. A method for running a development file, characterized in that: Applicable to resource clusters, including: Receive a resource scheduling request from a server; wherein the resource scheduling request is generated by the server based on a resource association request received from a client, and the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a pre-allocated schedulable resource, and a pre-configured operating environment pool in the schedulable resource; Creating a kernel process based on the schedulable resources indicated by the resource identifier carried in the resource scheduling request, and allocating an operating environment in the operating environment pool indicated by the resource identifier to the kernel process; When the kernel process receives a process call request carrying the file content of the target file, the target file is run based on the file content, and the file running result is obtained and returned; wherein the process call request is generated after the server establishes a communication connection with the kernel process based on the file identifier, and when a file running instruction for the target file is received, and is sent to the kernel process based on the communication connection.

11. The method according to claim 10, characterized in that Before receiving the resource scheduling request from the server, the method further includes: Receiving a resource configuration instruction from the server; wherein the resource configuration instruction is generated by the server based on a cluster identifier, a schedulable identifier, and at least one operating environment specification of a resource cluster carried in a resource creation instruction received from a client; Allocate schedulable resources based on the schedulable identifier carried by the resource configuration instruction; Based on the at least one operating environment specification carried by the resource configuration instruction, an operating environment pool is configured in the schedulable resources.

12. The method according to claim 11, characterized in that A data processing framework is deployed in the resource cluster, and the kernel process is created by a kernel service started by the data processing framework in the resource cluster; the kernel service is associated with a dependency installation location, and a plurality of dependency environments are installed on the dependency installation location; Then, configuring an operating environment pool in the schedulable resources based on the at least one operating environment specification carried by the resource configuration instruction includes: For the at least one operating environment specification, perform the following operations respectively: Establishing a basic environment based on an operating environment specification and at least one dependent environment inherited from the multiple dependent environments; According to a preset number, configuring a plurality of the basic environments for the kernel service in the schedulable resources; Add multiple configured basic environments as multiple running environments to the running environment pool.

13. The method according to claim 12, characterized in that The method further comprises: The kernel service is used to receive a dependency installation request for the operating environment pool; wherein the dependency installation request is generated by the server based on a received dependency addition instruction for the operating environment pool, and the dependency addition instruction carries an installation package identifier of a dependency environment; Using the kernel service to download the environment installation package of the dependent environment from a preset dependent environment source based on the installation package identifier carried in the dependent installation request; Based on the downloaded environment installation package, the one dependent environment is installed in the dependent installation location associated with the kernel service.

14. The method according to any one of claims 10 to 13, characterized in that: A data processing framework is deployed in the resource cluster, and the kernel process is created in the resource cluster using the data processing framework; the method further includes: Establishing a calling channel for using a third development language to call a function written in a second programming language used by the data processing framework; wherein the third development language is different from the second programming language; Then, the step of running the target file based on the file content to obtain the file running result includes: The kernel process is used to instruct the data processing framework to run the target file according to the file content based on the calling channel to obtain a file running result.

15. A device for running a development file, characterized in that: Applied to the server, including: The transceiver module is used to receive a resource association request from a client; wherein the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a schedulable resource pre-allocated from a resource cluster, and a running environment pool pre-configured in the schedulable resource; A processing module is used for scheduling the schedulable resources to create a kernel process based on the resource identifier, and allocating an operating environment in the operating environment pool to the kernel process; The processing module is also used to: establish a communication connection with the kernel process based on the file identifier; The transceiver module is also used for: upon receiving a file execution instruction for the target file, calling the kernel process based on the communication connection to execute the target file, and obtaining and returning a file execution result.

16. A device for running a development file, characterized in that: Applicable to resource clusters, including: The transceiver module is used to receive a resource scheduling request from a server; wherein the resource scheduling request is generated by the server based on a resource association request received from a client, and the resource association request carries: a file identifier of a target file and a resource identifier of a pre-configured resource; the pre-configured resource includes: a pre-allocated schedulable resource, and a pre-configured operating environment pool in the schedulable resource; Processing module: used for creating a kernel process based on the schedulable resources indicated by the resource identifier carried in the resource scheduling request, and allocating an operating environment in the operating environment pool indicated by the resource identifier to the kernel process; The processing module is also used to: when the kernel process receives a process call request carrying the file content of the target file, run the target file based on the file content, obtain and return the file running result; wherein the process call request is generated after the server establishes a communication connection with the kernel process based on the file identifier and when a file running instruction for the target file is received, and is sent to the kernel process based on the communication connection.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

18. A computer device, characterized in that: include: A memory for storing program instructions; A processor, configured to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 14 according to the obtained program instructions.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 14.

Images