Environment isolation method and apparatus based on spark computing engine, and device

Abstract

The present invention provides an environment isolation method based on a Spark computing engine. The method is used to perform, on the basis of a first instruction, environment isolation for a task running on a Spark computing engine, and at least comprises a modification step: before the task runs, creating and generating a first instance of a redefined ExternalCatalog class. In the modification step, the redefined ExternalCatalog class comprises one or more overridden interface functions related to a read or write operation in an original ExternalCatalog class. The technology provided by the present invention can improve the flexibility and scalability of environment isolation technology, thereby enabling better environment isolation for different tasks.

Description

A Spark computing engine-based environment isolation method, device and equipment Technical Field

[0001] The present invention relates to the field of data processing, and in particular to an environment isolation technology. Background Art

[0002] For work involving data processing, especially production and development work relying on big data, data security and flexibility are often paramount considerations. Environmental isolation technology can simultaneously meet the security and flexibility requirements of production and development work. On the one hand, environmental isolation technology isolates and stores different types of data in separate environments, thereby limiting access to data. On the other hand, by separating data in different environments, environmental isolation technology prevents failures in one environment from affecting data in other environments, thereby improving data reliability and availability. Furthermore, by separating data in different environments, environmental isolation technology makes it easier to manage and maintain environments, allowing them to be expanded or adjusted as needed, thereby increasing system flexibility and scalability.

[0003] Currently, existing technologies for isolating data environments can be categorized into two types: physical isolation and logical isolation. Physical isolation ensures the security of different data environments by isolating physical clusters. For example, the secondary environment and the primary environment are located in different physical clusters. This approach offers high security but limited flexibility. It also presents numerous challenges. For example, more physical clusters mean higher maintenance costs, more environment deployment, debugging, and release work, and inconvenient interaction between the primary and secondary environments.

[0004] Logical isolation can be achieved in a variety of ways, including virtualization and permissions management. While these methods can increase data flexibility to a certain extent, they still present challenges. For example, virtualization requires the creation of multiple virtual database instances, each running in an independent virtual environment. This provides better isolation but increases resource consumption and management complexity. Permission management can effectively control data access rights, but requires extensive management effort and is prone to security vulnerabilities caused by incorrect permission configuration.

[0005] Additionally, existing technologies also exist to achieve logical environment isolation by parsing and modifying task statements. For example, modifying a task's stored procedures to restrict access to specific data and operations. However, this approach has significant drawbacks: it requires different parsing and modification rules for tasks in different languages ​​and rules, resulting in low scalability and increased system complexity.

[0006] Therefore, how to address the shortcomings of the above-mentioned environmental isolation technology and provide a flexible and highly scalable environmental isolation technology is a technical problem that needs to be urgently solved in this field. Summary of the Invention

[0007] The present invention provides an environment isolation method, device and equipment based on the Spark computing engine, the purpose of which is to improve the flexibility and scalability of environment isolation technology so that it can better achieve environment isolation for different tasks.

[0008] In order to achieve the above-mentioned purpose, a technical solution adopted by the present invention is: an environment isolation method based on a Spark computing engine, the method is used to perform environmental isolation on tasks running on the Spark computing engine according to a first instruction, and at least includes a modification step: before the task runs, creating and generating a first instance of a redefined ExternalCatalog class; in the above-mentioned modification step, the redefined ExternalCatalog class includes rewriting one or more interface functions related to read operations or write operations in the original ExternalCatalog class, and the rewriting at least includes: modifying the first interface function related to the write operation so that the first interface function can divide the execution environment of the first object of the first interface function according to the first instruction; and / or modifying the second interface function related to the read operation so that the second interface function can divide the execution environment of the second object of the second interface function according to the second instruction; wherein, the execution environment includes at least a first environment and a second environment, and the first environment and the second environment are environmentally isolated.

[0009] In a preferred embodiment, the modification of the first interface function related to the write operation includes: judging the execution environment of the task according to the first instruction, and if the execution environment of the task is the first environment, executing the write operation; if the execution environment of the task is the second environment, modifying the first object so that it is executed in the second environment; wherein, the modification includes: at least modifying the original file name and / or storage path of the first object to the file name and / or storage path in the second environment; the file name includes the database name and / or data table name.

[0010] In a preferred embodiment, the modification of the second interface function related to the read operation includes: obtaining the second object according to the second instruction; judging the execution environment of the second object, and if the execution environment of the second object is the first environment, performing the read operation; if the execution environment of the second object is the second environment, judging whether the second object exists in the second environment; if the second object exists in the second environment, performing the read operation; if the second object does not exist in the second environment, modifying the second interface function so that the second object is found in the execution environment before performing the read operation.

[0011] In a preferred embodiment, obtaining the second object includes determining the type of the second object, and if the second object is temporary data, performing a read operation in an execution environment that creates the second object.

[0012] In a preferred embodiment, the creating of the redefined ExternalCatalog class includes: replacing the original ExternalCatalog class of the Spark computing engine with the redefined ExternalCatalog class through a reflection method.

[0013] In a preferred embodiment, before executing the modification step, the method further includes a configuration step for obtaining upstream data information of the task through static lineage resolution; the second instruction is determined based on the upstream data information.

[0014] In a preferred embodiment, after executing the modification step, the method further comprises a verification step for verifying whether the modification step will have an impact on the data of the first environment.

[0015] In order to achieve the above-mentioned purpose, another technical solution adopted by the present invention is: an environment isolation device based on a Spark computing engine, the device including an environment isolation module, which is used to perform environmental isolation on tasks running on the Spark computing engine according to a first instruction; the environment isolation module includes a first modification module, which is used to create and generate a first instance of a redefined ExternalCatalog class before the task runs; wherein, the redefined ExternalCatalog class includes rewriting one or more interface functions related to read operations or write operations in the original ExternalCatalog class; the first modification module includes a second modification module, which is used to modify the first interface function related to the write operation so that the first interface function can divide the execution environment of the first object of the first interface function according to the first instruction; and / or modify the second interface function related to the read operation so that the second interface function can divide the execution environment of the second object of the second interface function according to the second instruction; wherein, the execution environment includes at least a second environment and a first environment, and the second environment and the first environment are environmentally isolated.

[0016] In a preferred embodiment, the device includes a configuration module and a verification module, wherein the configuration module is used to obtain the upstream data information of the task through static lineage analysis before executing the modification step; and the verification module is used to verify whether the modification step will affect the data of the first environment after executing the modification step.

[0017] In order to achieve the above-mentioned purpose, another technical solution adopted by the present invention is: an electronic device, which includes a data processing device and a data storage device, and the data storage device stores a computer program. The computer program can be loaded and executed by the data processing device to implement the environment isolation method based on the Spark computing engine as described in any one of claims 1 to 7.

[0018] Compared with the existing technology, the advantages of the present invention are: (1) users are unaware of the method of implementing environment isolation in the present invention, and from the user's perspective, operations in different execution environments are always the same; (2) the technology provided by the present invention can adapt to all supported languages ​​of the Spark computing engine; (3) users are supported to select the execution environment for reading upstream data; (4) the technology provided by the present invention uses a customized metadata management service to enable all tasks developed using the Spark computing engine to read and write data in the environment selected by the user without the user modifying the code, and maintain a certain degree of connectivity, that is, the second environment can read the data of the first environment, making the development process more flexible. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] FIG1 is a schematic diagram of an environment isolation device based on a Spark computing engine provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following is a clear and complete description of the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0022] As used herein, "at least one" means one or more than one.

[0023] In this article, "first", "second", etc. are only used to distinguish different objects, rather than to describe a specific order.

[0024] In this document, terms such as "include", "comprising", "containing" and "having" are open-ended and indicate the presence of the described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0025] In this article, "Spark computing engine" refers to an open source big data processing engine development project initiated by the AMP Laboratory of the University of California, Berkeley; its currently disclosed versions are all applicable to the present invention, for example, the open source project spark with version number 3.3.0 uploaded by the Apache Spark community on the github platform on June 16, 2022; without changing the structure, function or operation mode of the necessary components / modules involved in the present invention, the Spark engine further developed based on the above version can also be applied to the present invention.

[0026] In this article, environment isolation is a practice in software development and information technology that isolates environments for different stages or purposes (such as development, testing, and production) from each other to ensure their independence and stability. Through environment isolation, developers can develop, test, and deploy code without affecting the primary environment. This helps reduce the impact of errors or failures caused by the development and testing process on the primary environment, while also improving system security and stability.

[0027] In this article, the execution environment refers to the hardware and software environment and configuration conditions that a task or program relies on during runtime, including the operating system, hardware resources, network environment, runtime libraries, and dependencies. These factors directly affect the execution efficiency, stability, and reliability of the task.

[0028] In this document, a write operation refers to an operation used to change the state or content of data, including operations such as modifying, adding, or deleting data.

[0029] In this article, read operations refer to operations used to obtain or retrieve data for further processing, analysis, or display in an application, including operations such as retrieving or obtaining data from a data source (such as a database, file system, etc.).

[0030] The Spark computing engine is a fast, versatile computing engine designed for large-scale data processing. It can be used to perform a wide range of operations, including SQL queries, text processing, and machine learning. Furthermore, the Spark computing engine provides multiple programming languages ​​and APIs for developing tasks tailored to specific needs, such as Scala, Python, Java, and SQL. However, the Spark computing engine itself does not directly provide environment isolation technology. Developers cannot implement development and production in different environments based on the Spark computing engine.

[0031] Based on this, the present invention provides an environment isolation technology based on the Spark computing engine. This technology can provide environment isolation for Spark-based task development and production without intruding into task code or parsing. Furthermore, this technology does not affect the Spark computing engine's operations.

[0032] Specifically, the Spark computing engine-based environment isolation technology provided by the present invention realizes environment isolation by modifying the execution logic of the task read and / or write operations in the Spark computing engine. On the one hand, the present invention realizes logical environment isolation by controlling the execution environment of the task read and / or write operations on data, that is, different data sources can be specified in different tasks to ensure the isolation of data in different environments. For example, the technology provided by the present invention can store production data in one database or file system and development data in another database or file system. The production environment and the development environment can access different data, and modifying the data in one environment will not affect the data in the other environment, thereby achieving logical isolation and avoiding data confusion or conflict. The present invention helps protect the security and integrity of production data through this method and provides a better development and testing environment. On the other hand, the present invention realizes the aforementioned control at the computing engine level, thereby ensuring that the technology provided by the present invention can be applied to tasks developed by various programming languages ​​​​without the need for adaptive adjustments for tasks of different types and languages.

[0033] More specifically, unlike existing technologies, the technology provided by the present invention implements isolation control by modifying the Spark computing engine's ExternalCatalog class to implement the execution environment for read and / or write operations. Common control methods in the prior art include controlling access rights, configuring data sources, setting environmental parameters, converting and filtering data, and so on. These existing control methods have many drawbacks. For example, they cannot be used in scenarios with large systems or complex data sources; they cannot guarantee the consistency of configurations across multiple tasks or systems; and re-setting the environment or permissions will affect the flexibility and scalability of the system. In contrast, the technology provided by the present invention can solve all of the aforementioned problems.

[0034] One embodiment of the present invention provides an environment isolation method based on a Spark computing engine. The method is used to isolate the environment of a task running on the Spark computing engine according to a first instruction, and at least includes a modification step: before the task runs, creating and generating a first instance of a redefined ExternalCatalog class; in the above modification step, the redefined ExternalCatalog class includes rewriting one or more interface functions related to read operations or write operations in the original ExternalCatalog class, and the rewriting at least includes: modifying the first interface function related to the write operation so that the first interface function can divide the execution environment of the first object of the first interface function according to the first instruction; and / or modifying the second interface function related to the read operation so that the second interface function can divide the execution environment of the second object of the second interface function according to the second instruction; wherein the execution environment includes at least the second environment and the first environment, and the second environment and the first environment are environmentally isolated.

[0035] In this embodiment, the Spark computing engine implements environmental isolation according to the first instruction. The first instruction is used to indicate the operating environment of the current task, for example, the second environment or the first environment. In an optional embodiment, the first instruction can be an input instruction received by the computing engine from the outside, such as an instruction input through a user interface, a command line, an API interface, etc.; at the same time, it can also be an instruction generated by other programs, scripts, tasks, etc., and passed to the computing engine through program calls, message queues, file transfers, etc. Specifically, after receiving the first instruction, the Spark computing engine executes a modification step. The modification step occurs before the task runs, and includes creating and generating a first instance of the redefined ExternalCatalog class. The creation of the redefined ExternalCatalog class includes creating a new ExternalCatalog class (referred to as the "redefined ExternalCatalog class" in the present invention) based on the original ExternalCatalog class of the Spark computing engine. The redefined ExternalCatalog class rewrites the interface functions in the original ExternalCatalog class of the Spark computing engine (including modifying one or more steps or methods in the interface functions) and replaces the execution of the original ExternalCatalog class. On this basis, it can be understood that the redefined ExternalCatalog class will be treated as the original ExternalCatalog class and used in the Spark computing engine, that is, it has the same or similar functions and working principles as the original ExternalCatalog class: an instance (first instance) is created through the redefined ExternalCatalog class, which initializes the relevant connection and configuration information for connecting to the external data storage system and loads metadata information from the external storage system. Once the instance is created and initialized, it can take effect within the life cycle of a Spark session (that is, the life cycle of a task) and be used by various parts of the Spark computing engine, including the execution of the current task.

[0036] In an optional embodiment, creating a redefined ExternalCatalog class includes: replacing the original ExternalCatalog class of the Spark computing engine with the redefined ExternalCatalog class through a reflection method. Specifically, the reflection method used in this embodiment allows the acquisition of class information, such as class fields, methods, constructors, etc., when the Spark computing engine performs operations, as well as operations such as dynamically calling class methods and creating object instances. In an optional embodiment, the reflection method can be implemented by a built-in function for implementing reflection functions in the ecosystem of the programming language. It can be understood that the present invention uses a reflection mechanism to modify or replace the original ExternalCatalog class of the Spark computing engine without modifying the source code, avoiding the step of determining the specific information of the class at compile time, and can dynamically load and replace classes as needed to meet specific needs or implement specific functions.

[0037] Because instances created by the ExternalCatalog class are used to execute tasks, the present invention modifies one or more interface functions in the ExternalCatalog class related to read or write operations to achieve environment isolation. That is, the first instance selects to execute different logic or operations based on the task execution environment indicated by the first instruction to meet the requirements of environment isolation. More specifically, the redefined ExternalCatalog class rewrites the original ExternalCatalog class by at least: modifying the first interface function related to the write operation so that the first interface function can divide the execution environment of the first object of the first interface function according to the first instruction; and / or modifying the second interface function related to the read operation so that the second interface function can divide the execution environment of the second object of the second interface function according to the second instruction. The second instruction is used to indicate the upstream data source required by the current task, such as the storage environment, path, or name of the upstream data of the task. In an optional embodiment, the second instruction can be an input instruction received by the computing engine from the outside, such as an instruction input through a user interface, command line, API interface, etc.; it can also be an instruction generated by other programs, scripts, tasks, etc. and transmitted to the computing engine through program calls, message queues, file transfers, etc. Specifically, there can be multiple execution environments. This embodiment determines the execution environment of a task based on the first instruction and sets the read and write operations of the task in the determined execution environment. It can be understood that the present invention can achieve logical environment isolation of tasks by controlling the read and write operations of the task.

[0038] Furthermore, in this embodiment, the execution environment includes at least a second environment and a first environment, wherein the second environment and the first environment are environmentally isolated. It can be understood that the environmental isolation between the first environment and the second environment is a static isolation, that is, the storage space corresponding to the first environment and the second environment is physically and / or logically isolated; further, the technology provided by the present invention can achieve environmental isolation for new data or existing data during dynamic task development. In an optional embodiment, the execution environment can be divided into multiple independent environments according to actual needs, for example, one or more second environments, and / or one or more first environments. In an optional embodiment, when the first environment is a production environment or a development environment, the second environment is a development environment or a production environment.

[0039] One embodiment of the present invention provides an environment isolation method based on the Spark computing engine. The method includes at least a modification step: before the task runs, creating and generating a first instance of a redefined ExternalCatalog class; in the modification step, the redefined ExternalCatalog class includes rewriting one or more interface functions related to read or write operations in the original ExternalCatalog class, and the rewriting includes at least modifying the first interface function related to the write operation so that the first interface function can divide the execution environment of the object of the first interface function according to the first instruction.

[0040] In an optional embodiment, the modification of the first interface function related to the write operation includes: judging the execution environment of the task according to the first instruction, if the execution environment of the task is the first environment, executing the write operation; if the execution environment of the task is the second environment, modifying the first object so that it is executed in the second environment; wherein the modification includes: at least modifying the original file name and / or storage path of the first object to the file name and / or storage path in the second environment; the file name includes the database name and / or data table name. Specifically, for modifying the first interface function related to the write operation, it is first necessary to determine the execution environment of the task. In an optional embodiment, the first environment can be set to the default environment, that is, the environment in which the task is executed when no modification is made to the Spark computing engine (ExternalCatalog class). At this time, when the task execution environment indicated by the first instruction is the first environment, the write operation is executed normally; when the task execution environment indicated by the first instruction is the second environment, the write operation is modified so that the first object of the first interface function is executed in the second environment. In an optional embodiment, the modification of the write operation can optionally modify the original file name and / or storage path of the first object to the file name and / or storage path in the second environment, so that the first object can be executed in the second environment and isolated from the data in the first environment. Among them, modifying the file name is used to isolate metadata, and modifying the storage path is used to isolate data. The first object can be an object that needs to be written to or perform a write operation in the execution environment, such as a data table. In an optional embodiment, the modification of the file name can be selected to modify the database name and / or modify the data table name. The database name is the upper-level file system of the data table name. Modifying any one or both of them can implement the present invention.

[0041] It should be understood that setting the first environment or the second environment as the default environment does not affect the implementation of the present invention.

[0042] One embodiment of the present invention provides an environment isolation method based on the Spark computing engine. The method includes at least a modification step: before the task runs, creating and generating a first instance of a redefined ExternalCatalog class; in the modification step, the redefined ExternalCatalog class includes rewriting one or more interface functions related to read or write operations in the original ExternalCatalog class, and the rewriting includes at least modifying a second interface function related to the read operation so that the second interface function can partition the execution environment of the second interface function's object according to a second instruction.

[0043] In an optional embodiment, modifying the second interface function associated with the read operation includes: obtaining the second object according to the second instruction; determining the execution environment of the second object, and if the execution environment of the second object is the first environment, performing the read operation; if the execution environment of the second object is the second environment, determining whether the second object exists in the second environment; if the second object exists in the second environment, performing the read operation; and if the second object does not exist in the second environment, modifying the second interface function so that the second object is found in the execution environment before performing the read operation. Specifically, in this embodiment, obtaining the second object according to the second instruction. In an optional embodiment, obtaining the second object includes obtaining the storage path, file name, and / or data table name of the second object. For the second interface function, the second object may include one or more objects that need to be read or have a read operation performed, and these objects may be distributed across different execution environments. Furthermore, in addition to considering the actual storage location of the second object, developers may also need to modify the upstream data source to adapt to the execution environment of the current task. For example, for tasks executed and calculated in a development environment, developers may desire that the upstream data source also be stored in the development environment to avoid contaminating data in the production environment. Therefore, the step of modifying the second interface function associated with the read operation in this embodiment is designed to accommodate the selection of the task's execution environment.

[0044] Specifically, the first environment is considered the default environment for task execution, and the second instruction obtains the upstream data of the task (i.e., the second object) based on the metadata system of the first environment. It is easy to understand that due to the environment of the metadata system, the second object naturally includes the upstream data existing in the first environment. At this time, for the second object selected as the first environment, there is no need to determine whether the second object exists before the read operation can be performed. To modify the execution environment of the upstream data source, the second instruction includes the selection of the upstream data in the metadata system of the default environment (first environment) corresponding to the second environment. For example, for data table A stored only in the first environment, the second instruction also provides the selection of data table A in the second environment. If the developer selects the current task to read data table A from the second environment, the second object is data table A in the second environment (different from data table A in the first environment). At this time, for the read operation in the second environment, this embodiment needs to determine whether the second object exists in the second environment. In addition, the second instruction includes the selection of upstream data stored only in the metadata system of the non-default environment (second environment). For example, for data table B (second object) stored only in the second environment, the developer cannot select to perform a read operation in the first environment based on the second instruction. It is understandable that in this case, in order to ensure that the data in the first environment is not contaminated, this embodiment will not pass the upstream data stored only in the second environment to the first environment, and developers cannot choose to perform read operations in the first environment.

[0045] In an optional embodiment, it is possible to determine whether the second object exists by determining whether the corresponding file name (including database name and / or data table name) exists in the storage path of the second object. It should be noted that the present invention can identify the execution environment by the storage path or file name; taking the example of distinguishing the execution environment by the storage path or database name, the same data table name may exist in different execution environments, while the same data table name does not exist in the same execution environment. In an optional embodiment, the execution environment can also be determined by the or data table name. More specifically, this embodiment determines the execution environment of each object in the second object respectively. If the execution environment of the second object is the first environment, a read operation is performed; if the execution environment of the second object is the second environment, it is determined whether the second object exists in the second environment; if the second object exists in the second environment, a read operation is performed; if the second object does not exist in the second environment, the read operation is modified so that the second interface function finds the second object in the execution environment and then performs the read operation. In an optional embodiment, making the second interface function find the second object in the execution environment includes copying and transferring the second object to the second environment after finding it in the first environment. At this time, the read operation for the second object can be completed in the second environment.

[0046] In an optional embodiment, the acquisition of the second object includes determining the type of the second object, and if the second object is temporary data, performing a read operation in the execution environment in which the second object is created. Taking into account that temporary data may exist in different execution environments at the same time, and its value may be modified or updated in a certain execution environment, this embodiment specifically sets a read operation step for temporary data, that is, performing a read operation in the creation environment of the temporary data to ensure that the write operation of the temporary data can be read by subsequent read operations. It can be understood that after performing the read operation on the temporary data, there is no need to determine the execution environment of the second object and perform the read operation.

[0047] One embodiment of the present invention provides an environment isolation method based on a Spark computing engine. The method is used to isolate the environment of a task running on the Spark computing engine according to a first instruction, and at least includes the following modification step: before the task runs, creating and generating a first instance of a redefined ExternalCatalog class.

[0048] In an optional embodiment, before executing the modification step, the method further includes a configuration step for obtaining the upstream data information of the task through static lineage analysis; the second instruction is determined based on the upstream data information. Specifically, in the configuration step, the developer can customize the selection of reading upstream data from the first environment and / or from the second environment as the input of the task. The selection of upstream data will serve as the first instruction for executing the subsequent modification step. In an optional embodiment, the upstream data information of the task can be obtained through static lineage analysis. The method of static lineage analysis includes: step S1: receiving the target task, which is a table-based data processing task; step S2: registering a listener on the Spark computing engine; step S3: adding a logical plan node replacement rule in the parser of the Spark computing engine, the logical plan node replacement rule is used to replace each logical plan node involving at least one of data reading, creation, writing or deletion with a corresponding virtual node, and is set to be executed after the parsing phase of the parser is completed, the virtual node only reads and / or creates metadata when performing read and create operations, and only writes other metadata other than the metadata of the original data source of the target task when performing write and delete operations. Insertion and / or deletion, and each of the virtual nodes and the logical plan node it replaces have the same impact on data lineage; step S4: starting the execution of the target task on the Spark computing engine; step S5: based on the logical plan event obtained by the listener, extracting lineage-related information, constructing a table relationship tree, an attribute tree, and optionally constructing a function tree; step S6: traversing the table relationship tree, the attribute tree, and optionally traversing the function tree when the function tree has been constructed, to establish the data lineage of the target task, the data lineage of the target task is used to reflect the lineage relationship between the tables and / or fields involved in the target task; wherein, step S2 and step S3 can be interchanged in order, and any one of them can be advanced to before step S1. Among them, the events generated by the Spark computing engine that the listener can monitor include events containing logical plans ("logical plan events"). Specifically, the listener used in this embodiment can monitor events containing query plans (referring to QueryExecution in the Spark computing engine). The "query plan" includes a complete parsed logical plan ("logical execution plan"), and the event containing the query plan is the logical plan event. Registering the listener on the Spark engine allows data lineage to be established in subsequent steps based on the logical plan events obtained through monitoring.The parser in the Spark computing engine will first add a corresponding logical plan node for each operation in the task. Each logical plan node represents a specific logical operation. The nodes are connected through a parent-child relationship to form a complete logical execution plan with a tree structure, which is used to represent the sequence of logical operations (i.e., the dependency relationship between logical operations). Since both registering the listener and adding the logical plan node replacement rule need to take effect only when the Spark computing engine executes the target task, and registering the listener and adding the logical plan node replacement rule are independent of each other, and between the two and the receiving target task, there is no necessary order relationship. Therefore, in the static parsing method of this embodiment, step S2 and step S3 can be interchanged, and either one can be advanced before step S1.

[0049] In an optional embodiment, after executing the modification step, the method further includes a verification step for verifying whether the modification step affects the data in the first environment. Specifically, the verification step is configured to verify metadata associated with the task to ensure that the modification step does not contaminate the data in the first environment. In an optional embodiment, the verification can be performed using pattern matching.

[0050] One embodiment of the present invention provides an environment isolation device based on a Spark computing engine, the device including an environment isolation module for performing environment isolation on a task running on the Spark computing engine according to a first instruction; the environment isolation module includes a first modification module for creating and generating a first instance of a redefined ExternalCatalog class before the task runs; in the above modification step, the redefined ExternalCatalog class includes rewriting one or more interface functions related to read operations or write operations in the original ExternalCatalog class; the first modification module includes a second modification module for modifying the first interface function related to the write operation so that the first interface function can divide the execution environment of the object of the first interface function according to the first instruction; and / or modifying the second interface function related to the read operation so that the second interface function can divide the execution environment of the object of the second interface function according to the second instruction; wherein the execution environment includes at least a second environment and a first environment, and the second environment and the first environment are environmentally isolated.

[0051] Specifically, the second modification module is used to determine the execution environment of the task according to the first instruction. If the execution environment of the task is the first environment, the write operation is performed; if the execution environment of the task is the second environment, the first object is modified so that it is executed in the second environment. The modification includes at least changing the original file name and / or storage path of the first object to the file name and / or storage path in the second environment; the file name includes the database name and / or data table name. The second modification module is also used to obtain the second object according to the second instruction; determine the execution environment of the second object; if the execution environment of the second object is the first environment, the read operation is performed; if the execution environment of the second object is the second environment, the read operation is performed; if the second object exists in the second environment, the read operation is performed; if the second object does not exist in the second environment, the modification is performed so that the second interface function finds the second object in the execution environment before performing the read operation. In an optional embodiment, the second modification module can optionally determine the type of the second object when obtaining the second object for the read operation. If the second object is temporary data, the read operation is performed in the execution environment where the second object was created.

[0052] In an optional embodiment, the first modification module is used to replace the original ExternalCatalog class of the Spark computing engine with the redefined ExternalCatalog class through a reflection method.

[0053] In an optional embodiment, the device includes a configuration module and a verification module, wherein the configuration module is used to obtain upstream data information of the task through static lineage analysis before executing the modification step; and the verification module is used to verify whether the modification step will affect the data of the first environment after executing the modification step.

[0054] One embodiment of the present invention provides an electronic device, which includes a data processing device and a data storage device, wherein the data storage device stores a computer program, and the computer program can be loaded and executed by the data processing device to implement the environment isolation method based on the Spark computing engine as described in any embodiment provided by the present invention.

[0055] The above embodiments describe the present invention or an aspect of the present invention, and these embodiments may be combined arbitrarily. On this basis, in order to more clearly reflect the process of the present invention, the following embodiments are illustrative in combination with specific cases.

[0056] As shown in Figure 1, the environment isolation device based on the Spark computing engine provided in this embodiment includes a configuration module, an environment isolation module (including a modification module) and a verification module. Among them, the configuration module is used to obtain the upstream data information of the task through static lineage resolution, the modification module is used to create and generate an instance of the redefined ExternalCatalog class, and the verification module is used to verify whether the modification module will affect the data of the first environment. In this embodiment, the system's default execution environment is the first environment, and the optional execution environment includes the second environment. This embodiment implements environmental isolation of the first environment and the second environment through an environment isolation method based on the Spark computing engine.

[0057] Specifically, developers can select the execution environment (i.e., the first instruction) for a specific task, for example, select the task to run in the second environment or the first environment. Generally speaking, in the test development stage, developers will give priority to selecting the task to run in the development environment to debug or develop the task code and function. After the task code is finally determined, it will be set in the production environment to perform operations on real data. Accordingly, the data in the development environment can be real data or test data for task development. According to the technology provided by the present invention, since the production environment and the development environment are isolated from each other, the calculation and modification of data will not affect the real data in the first environment when the development environment debugs the task code. In the meantime, when it is necessary to use real data for task testing, the development environment is easy to interact with the first environment, for example, the data in the production environment is copied and transferred to the development environment for use in task testing. As can be seen, the environment isolation technology based on the Spark computing engine provided by the present invention has both guaranteed the isolation effect between different environments and possessed the flexibility of interaction between different environments.

[0058] More specifically, in this embodiment, after the developer selects the execution environment of the task, the configuration module calls the static lineage analysis method to obtain the upstream data information of the task. In an optional embodiment, after the static lineage analysis is triggered, the upstream data information calculated by it can be displayed to the developer. Furthermore, the developer can choose to read the upstream data (i.e., the second instruction) from the first environment or the second environment based on the upstream data information provided by the configuration module. At this point, the workflow of the configuration module is completed and enters the modification module.

[0059] As mentioned above, the main task of the modification module is to execute the modification step, that is, before the task is run, create and generate an instance of the redefined ExternalCatalog class (the first instance). For the avoidance of doubt, the modification module will create a redefined ExternalCatalog class in each task execution cycle, and inject the modification logic according to the modification step (the modification logic can be understood as the logical method for the present invention to achieve environmental isolation). The modification logic injected into each task cycle is the same. Since all tasks submitted to the Spark computing engine will pass through the ExternalCatalog class when interacting with the metadata system, that is to say, the modification logic contained in the modification step of the present invention is universal, that is, it can adapt to all development languages ​​supported by the Spark computing engine. At the same time, if the rules for environmental isolation need to be adjusted or modified, it is only necessary to change the modification logic without having to modify it separately for each task or each situation.

[0060] In this embodiment, the modification module replaces the original ExternalCatalog class of the Spark computing engine with a redefined ExternalCatalog class through reflection, and uses the redefined ExternalCatalog class to generate a first instance, through which the task is executed. Compared to the original ExternalCatalog class, the redefined ExternalCatalog class modifies one or more interface functions related to read or write operations, thereby ensuring that the read and write operations of the task are executed in the correct execution environment. For example, the more important interface functions in the ExternalCatalog class include:

[0061] requireTableExists->Judge whether the table exists

[0062] createTable->Create table

[0063] getTable->read table

[0064] dropTable->Delete table

[0065] loadPartition->Read table partition information

[0066] createDatabase->Create a database

[0067] In an optional embodiment, since the ExternalCatalog class encapsulates all interface functions of the Spark computing engine that interact with the metadata system, in order to achieve the best effect of environmental isolation, all interface functions in the ExternalCatalog class can be modified.

[0068] Specifically, taking the above-mentioned “createTable->create table” and “getTable->read table” as an example, this embodiment illustrates the logic of the modification steps in detail and exemplarily.

[0069] The "createTable" interface function is used to create a data table and is related to write operations. The modification module modifies the interface function so that the interface function can divide the execution environment of the object of the interface function according to the first instruction. The operation logic of the modified "createTable" interface function is:

[0070] The task execution environment is determined according to the first instruction. If it is the first environment, the write operation is performed. It should be noted that the write operation performed here is the write operation defined for the "createTable" interface function in the original ExternalCatalog class. The modification steps do not modify this part of the write operation.

[0071] If the execution environment is the second environment, the write operation is modified so that it is executed in the second environment. The modification includes changing the original file name and storage path of the first object of the first interface function to the file name and storage path in the second environment. For example, the original database name "new.table" is changed to the database name "dev_news.table" in the second environment, and the original storage path "bucket1 / Database / news / table" is changed to the storage path "bucket2 / Database / dev_news / table" in the second environment. In alternative embodiments, other tag names may be used to distinguish execution environments.

[0072] The "getTable" interface function is used to read the data table and is related to the read operation. The modification module modifies the interface function so that the interface function can divide the execution environment of the object of the interface function according to the second instruction. Comparatively, the read operation is more complicated than the write operation because for the write operation, one calculation will only include one write operation. Therefore, as long as the execution environment of the task is determined, the execution environment of the write operation can be determined. For the read operation, a calculation may read multiple data sources, some of which may come from the first environment and some from the second environment, and may even include temporary data (data temporarily generated or used during the calculation or processing process). Therefore, the modification logic of the read operation is more complicated than the write operation. The operation logic of the modified "getTable" interface function is:

[0073] According to the upstream data source selected by the developer (second instruction), obtain the second object of the "getTable" interface function;

[0074] Determine the type of the second object. If the second object is temporary data, proceed to the temporary data read operation below.

[0075] Determine the execution environment of the second object. If the execution environment of the second object is the first environment, perform a read operation. Note that the write operation performed here is the read operation defined for the "getTable" interface function in the original ExternalCatalog class. The modification step does not modify this read operation.

[0076] If the execution environment of the second object is the second environment, determining whether the second object exists in the second environment; in an optional embodiment, determining whether the second object exists may be performed based on the storage path and file name of the second object;

[0077] If the second object exists in the second environment, performing a read operation;

[0078] If the second object does not exist in the second environment, the "getTable" interface function is modified so that the second object is found in the execution environment before the read operation is performed. After the second object is found in the execution environment, this embodiment copies the second object to the second environment, and then performs the read operation in the second environment, thereby avoiding the data used for development from contaminating the data in the first environment.

[0079] Temporary data read operation: perform a read operation in the execution environment where the second object is created. Specifically, taking temporary data table B as an example, temporary data table B reads data from data table A and writes it to B, and then reads data from B and writes it to data table C. Among them, the process of writing from A to B is to write the data to table B of the second environment, and the process of writing from B to C is to read from table B in the first environment and then write it to C. In order to ensure that the most recently generated data can be obtained, this embodiment sets this temporary data read operation so that the read operation on the temporary data is consistent with the creation environment of the temporary data, ensuring that the data written when the table is created can be read by subsequent read operations. It can be understood that under normal circumstances, in order to ensure the validity and consistency of temporary data, the creation environment of temporary data is consistent with the execution environment of the task.

[0080] After executing the modification step, the verification module of this embodiment calls the pattern matching method to verify the modified metadata to ensure that the table name, file name, storage path, etc. will not affect the data in the first environment.

[0081] It can be seen that the technology provided by the present invention customizes the metadata management service class of the computing engine, so that all tasks developed using the computing engine can read and write data according to the environment selected by the user without the user modifying the code, and maintain a certain connectivity, that is, the second environment can read the data of the first environment, thereby improving the flexibility of development.

[0082] Using any of the embodiments provided by the present invention, the effect of environmental isolation can be achieved. After applying the embodiments provided by the present invention to the Spark computing engine, the developer selects an execution environment (e.g., a development environment, the first instruction), triggers static lineage resolution to display all upstream data tables for the current task, and the developer further selects an environment for the upstream data tables (the second instruction). If the default execution environment is the production environment, then by default, only tables in the production environment are read.

[0083] In this embodiment, take the task developed by the developer to read data from the three tables database_a.table_a, database_b.table_b, and database_c.table_c and write data into the database_d.table_d table in the development environment as an example. If at this time the database_a.table_a, database_b.table_b, and database_c.table_c tables all exist in the production environment, that is, the database_a.table_a, database_b.table_b, database_c.table_c, and database_d.table_d tables do not exist in the development environment, and the developer chooses to read the database_a.table_a and database_b.table_b tables from the production environment and the database_c.table_c table from the development environment.

[0084] According to the technology provided by this invention, since the database_a.table_a and database_b.table_b tables exist in the production environment, no additional processing is required and the corresponding read operations can be performed directly. However, when reading the database_c.table_c table from the development environment, this embodiment first determines whether the corresponding table exists in the development environment. If not, it also determines whether the corresponding table exists in the production environment. In this embodiment, since the database_c.table_c table does not exist in the development environment, this embodiment imports the database_c.table_c table from the production environment to the development environment.

[0085] After the above import is completed, the upstream data table information selected by the developer will be passed to the context of the Spark computing engine through the redefined ExternalCatalog class. The redefined ExternalCatalog class will obtain this information from the context of the Spark computing engine during subsequent judgments.

[0086] After the task begins executing, when a table read operation is required (one of the read operations, which is used as an example in this embodiment), the getTable method (interface function) in the redefined ExternalCatalog class will be triggered. The getTable method switches the table to the naming rules of the corresponding environment according to the logical rules described in the embodiment of the present invention, and modifies the original reading from database_a.table_a, database_b.table_b, and database_c.table_c to reading from database_a.table_a, database_b.table_b, and dev_database_c.table_c (this embodiment uses the example of adding a "dev_" prefix to indicate the development environment). When a table write operation is executed (one of the write operations, which is used as an example in this embodiment), the createTable method in the redefined ExternalCatalog class will be triggered, and database_d.table_d will be modified to dev_database_d.table_d.

[0087] After the task is successfully executed, two tables, dev_database_c.table_c and dev_database_d.table_d, are added to the development environment. In an alternative embodiment, the dev_database_d.table_d generated by this task can be directly used to continue developing downstream tasks as needed, without relying on the production environment (the database_d.table_d table does not exist in the first environment at this time).

[0088] The present invention has been introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The above implementation description is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. Changes and improvements to the present invention will be possible without exceeding the concept and scope specified in the appended claims. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. An environment isolation method based on Spark computing engine, characterized in that: The method is used to isolate the environment of tasks running on the Spark computing engine according to the first instruction, and at least includes the following modification steps: Before the task is run, creating and generating a first instance of the redefined ExternalCatalog class; In the above modification step, the redefined ExternalCatalog class includes rewriting one or more interface functions related to read operations or write operations in the original ExternalCatalog class, and the rewriting includes at least: Modifying a first interface function related to the write operation so that the first interface function can divide the execution environment of the first object of the first interface function according to the first instruction; and / or Modifying a second interface function related to the read operation so that the second interface function can divide the execution environment of the second object of the second interface function according to the second instruction; The execution environment includes at least a first environment and a second environment, and the first environment and the second environment are isolated from each other.

2. The method according to claim 1, characterized in that The first interface function related to modifying the write operation includes: determining an execution environment of the task according to the first instruction, and executing the write operation if the execution environment of the task is the first environment; If the execution environment of the task is the second environment, the modification is performed so that the first object is executed in the second environment; wherein the modification includes: at least changing the original file name and / or storage path of the first object to the file name and / or storage path in the second environment; the file name includes the database name and / or data table name.

3. The method according to claim 1, characterized in that The second interface function related to modifying the read operation includes: According to the second instruction, obtain the second object; Determining the execution environment of the second object, and if the execution environment of the second object is the first environment, performing the read operation; if the execution environment of the second object is the second environment, determining whether the second object exists in the second environment; If the second object exists in the second environment, the read operation is performed; if the second object does not exist in the second environment, the second interface function is modified so that the second object is found in the execution environment before the read operation is performed.

4. The method according to claim 3, characterized in that The acquiring of the second object includes determining the type of the second object, and if the second object is temporary data, performing a read operation in an execution environment for creating the second object.

5. The method according to claim 1, wherein The creation of the redefined ExternalCatalog class includes: The original ExternalCatalog class of the Spark computing engine is replaced with the redefined ExternalCatalog class through reflection.

6. The method according to any one of claims 1 to 5, characterized in that Before executing the modification step, the method further includes a configuration step for obtaining upstream data information of the task through static lineage analysis; the second instruction is determined based on the upstream data information.

7. The method according to any one of claims 1 to 5, characterized in that After executing the modification step, the method further includes a verification step for verifying whether the modification step will have an impact on the data of the first environment.

8. An environment isolation device based on Spark computing engine, characterized in that: The device includes an environment isolation module for performing environment isolation on tasks running on the Spark computing engine according to a first instruction; The environment isolation module includes a first modification module for creating and generating a first instance of the redefined ExternalCatalog class before the task is run; The redefined ExternalCatalog class includes rewriting one or more interface functions related to read operations or write operations in the original ExternalCatalog class; The first modification module includes a second modification module for modifying a first interface function related to the write operation so that the first interface function can divide the execution environment of a first object of the first interface function according to the first instruction; and / or modifying a second interface function related to the read operation so that the second interface function can divide the execution environment of a second object of the second interface function according to the second instruction; The execution environment includes at least a first environment and a second environment, and the first environment and the second environment are isolated from each other.

9. The device according to claim 8, characterized in that The device includes a configuration module and a verification module, wherein: The configuration module is used to obtain upstream data information of the task through static lineage analysis before executing the modification step; The verification module is used to verify whether the modification step will affect the data of the first environment after the modification step is executed.

10. An electronic device, characterized in that: The electronic device includes a data processing device and a data storage device, wherein the data storage device stores a computer program, and the computer program can be loaded and executed by the data processing device to implement the environment isolation method based on the Spark computing engine according to any one of claims 1 to 7.

Images