Large model data cleaning environment construction method and system based on dynamic packaging

Abstract

The invention relates to the technical field of data cleaning, in particular to a large model data cleaning environment construction method and system based on dynamic packaging, and the method comprises the following steps: dynamically constructing a basic operation environment package, dynamically constructing a service application package, automatically deploying a cluster environment, and dynamically loading and releasing a node-level environment. The method has the beneficial effects that the basic operation environment is automatically constructed according to the development language and the version thereof through a dynamic packaging mode, so that different language environments can be adapted, and the flexibility and the expandability of the system are improved. Meanwhile, a plurality of packaged basic operation environments are isolated from one another, and a plurality of data cleaning tasks do not influence one another.

Description

Technical Field

[0001] The present invention relates to the technical field of data cleaning, and in particular, to a method and system for constructing a large model data cleaning environment based on dynamic packaging. Background Technique

[0002] In the field of big data processing technology, data cleaning is an important link, and various programming languages and their related development packages are usually required. With the rapid development requirements of large models, the data cleaning environment has become diversified and complex. For example, in the data cleaning process of large models, it may involve multiple programming languages such as Java, Python, and Scala, as well as dependencies between different versions. In the field of software engineering, software development and deployment are important links. In this process, various development tools and environments are usually required. With the development of software engineering, software development and deployment have also become diversified and complex. For example, in order to ensure that the software can run properly in different environments, it is necessary to manage the dependencies of the software. The traditional dependency management method is to lock a fixed version of a language, including the development packages of the dependencies are also fixed. However, this method has some problems. For example, it cannot adapt to the differences between different environments and is prone to environmental compatibility problems.

[0003] In the existing technology, the automatic pipeline method is usually adopted to reduce the workload of setting up the data cleaning environment and the compatibility problem of dependency packages. Specifically, it is to use automated tools to automatically install and configure the running environment and dependency packages required for data cleaning. The advantage of this method is that it can reduce manual intervention and improve work efficiency. However, there are still some problems in the existing technology when dealing with the data cleaning environment. First of all, the existing technology usually locks a fixed version of a development language, including the development packages of the dependencies are also fixed. This method cannot adapt to the differences between different language environments and is not conducive to the introduction and application of new technology development packages. Secondly, the existing technology needs to face numerous running nodes, and it is also necessary to install the running environment and its changing dependency packages respectively, and the workload is complex and error-prone. Finally, although the existing technology uses an automatic pipeline, it cannot solve the compatibility problem of dependency packages. For example, there may be compatibility problems between different versions of dependency packages, resulting in the inability to execute data cleaning tasks normally. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for constructing a large model data cleaning environment based on dynamic packaging to solve the problems proposed in the above background technique.

[0005] To achieve the above purpose, the present invention provides the following technical solution: A method for constructing a large model data cleaning environment based on dynamic packaging, including the following steps:

[0006] Dynamically build the basic operating environment package: For the target language and its version, generate a compressed file containing the language runtime, basic dependency libraries, and development packages through a package management tool, supporting multi-language and multi-version environment isolation;

[0007] Dynamically build the business application package: Based on business requirements, package the cleaning logic code and its third-party dependency libraries into an independent compressed file, decoupled from the basic environment package;

[0008] Automatically deploy the cluster environment: Upload the two types of compressed packages to the cluster management node through the Spark client and dynamically bind them to the job running environment;

[0009] Dynamically load and release the node-level environment: Load the specified environment package as needed in the working node memory, and automatically clean up resources after the job ends, realizing the coexistence of multi-versions of the environment and zero local dependencies.

[0010] Preferably, the process of building the basic operating environment package includes a language version intelligent adaptation mechanism. By parsing the language version parameters in the task configuration, automatically call the corresponding package management tool to generate the environment package, where: the Python environment uses conda to manage multi-version dependencies; the Java / Scala environment builds a versioned dependency tree through Maven / SBT; after the environment package is generated, hash verification is used to ensure cross-node consistency.

[0011] Preferably, the construction of the business application package includes a dynamic dependency resolution and conflict detection mechanism. The specific steps are as follows:

[0012] a) Scan the import statements of the cleaning script and the build configuration file;

[0013] b) Generate a dependency relationship graph and detect version conflicts;

[0014] c) Generate an application package containing precise version dependencies through virtual environment isolation technology;

[0015] d) Support injecting business customization parameters through a configuration file to achieve environment reuse.

[0016] Preferably, the process of deploying the cluster environment adopts a two-layer distribution architecture, including:

[0017] Client pre-verification: Submit a test job in Local mode to verify the integrity of the environment package and the correctness of the business logic;

[0018] Cluster-level distribution: Use the --deploy-mode cluster parameter of Spark Yarn to push the environment package to all working nodes and specify the loading path through the spark.archives parameter;

[0019] Runtime binding: Force the specified version through environment variables such as PYSPARK_PYTHON / JAVA_HOME, and combine the JVM class loader isolation to achieve multi-environment parallelism.

[0020] Preferably, the dynamic loading and release mechanism includes the following technical features:

[0021] a) Based on the Spark Application lifecycle management, each cleaning task runs as an independent Application, with an exclusive JVM instance and an in-memory space for the environment package.

[0022] b) Pre-load the environment package into the node memory through the distributed caching mechanism to reduce disk I / O overhead.

[0023] c) Implement a resource recovery listener to automatically trigger the unloading of the environment package and release memory resources when the Application terminates.

[0024] d) Support dynamically adjusting the environment package caching policy through configuration parameters to balance resource utilization and job startup speed.

[0025] Preferably, it includes:

[0026] Client node: Used to dynamically construct a language version-specific basic runtime environment package and business function application package, and upload them to the cluster after local verification.

[0027] Cluster management node: Receive and store the uploaded environment packages and application packages, and be responsible for distributing the packages to the worker nodes.

[0028] Worker node: Dynamically load the specified environment package to execute data cleaning tasks, and automatically clean up resources after the tasks are completed.

[0029] The system supports multi-language and multi-version environment isolation, and realizes zero-dependency installation on the running nodes through the package binding mechanism to ensure environment consistency and compatibility.

[0030] Preferably, the client node includes an intelligent packaging module, which specifically implements the following functions:

[0031] a) Environment-aware construction: Parse the language and version parameters in the task configuration, and call the corresponding toolchain to generate the environment package. Use conda for Python and Maven for Java / Scala.

[0032] b) Dependency conflict detection: Detect version conflicts by constructing a dependency relationship graph, and generate an application package containing precise version dependencies.

[0033] c) Localized pre-verification: Submit a test job through the Spark Local mode to verify the integrity of the environment package and the correctness of the business logic.

[0034] d) Cluster deployment trigger: After verification, upload the two types of compressed packages to the cluster management node through the Spark Yarn mode.

[0035] Preferably, the cluster management node implements a two-layer environment distribution mechanism:

[0036] Global storage layer: Use a distributed file system to store all version environment packages, and establish a hash index to ensure cross-node consistency;

[0037] Dynamic push layer: Trigger automatic distribution through the --deploy-mode cluster parameter, and push the specified environment package to the target worker node, supporting dynamic combination of base packages and application packages according to job requirements;

[0038] The environment package distribution process includes a version verification mechanism to ensure that the environment loaded by the worker node is strictly consistent with the client build version.

[0039] Preferably, when the worker node has an environment isolation runtime, it includes the following technical features:

[0040] a) Memory-level loading: Specify the compressed package path through the spark.archives parameter, and cooperate with the PYSPARK_PYTHON / JAVA_HOME variable to force the loading of the specified version environment;

[0041] b) JVM instance isolation: Each data cleaning job runs as an independent Application, with a dedicated JVM instance and class loader, supporting parallel operation of multiple version environments;

[0042] c) Automatic resource recovery: Register a job lifecycle listener to trigger the unloading of the environment package when the Application terminates, releasing memory resources;

[0043] d) Cache optimization strategy: Support configuring the environment package cache threshold to balance resource utilization and job startup speed.

[0044] Preferably, the packaging module is designed using the strategy pattern and includes the following core components:

[0045] IPackage interface: Define a unified packaging specification, including the doRuntimeEnvPackage() and doApplicationPackage() methods;

[0046] Language-specific implementation classes:

[0047] PythonPackage: Create a virtual environment through conda and generate a tar package containing the specified Python version and dependencies;

[0048] Java Package: Build a zip package containing the specified JDK version and JAR dependencies through Maven;

[0049] Scala Package: Generate a compressed package containing the Scala compiler and library dependencies through SBT;

[0050] Each implementation class contains a version parser, which supports extracting language version parameters from the task configuration and mapping them to the corresponding basic environment template.

[0051] Compared with the prior art, the beneficial effects of the present invention are:

[0052] The method and system for constructing a large model data cleaning environment based on dynamic packaging proposed by the present invention automatically constructs a basic operating environment according to the development language and its version through dynamic packaging, which can adapt to different language environments and improve the flexibility and scalability of the system. At the same time, the multiple basic operating environments packaged are isolated from each other, and multiple data cleaning tasks do not affect each other.

[0053] By dynamically constructing application packages on the basic operating environment and automatically uploading these packages to the cluster management node before cleaning operations, the cluster management node distributes them to each running node for decompression and dynamically specifies the current running environment to bind these packages. In this way, all running nodes do not need to install these operating environments and their packages additionally, and all dependent operating environments and packages have been automatically packaged and bound. This method effectively solves the compatibility problem of dependent packages and avoids the problem that the data cleaning task cannot be executed normally due to possible compatibility problems of different versions of dependent packages.

[0054] Through the dynamic packaging and automatic distribution mechanism, all running nodes do not need to manually install the operating environment and its packages, and all dependent operating environments and packages have been automatically packaged and bound, which greatly reduces manual intervention and workload and improves work efficiency. At the same time, since all dependent operating environments and packages have been automatically packaged and bound, errors and inconsistencies that may occur due to manual installation are also avoided. Brief Description of the Drawings

[0055] Figure 1 It is the system block diagram of the present invention. Detailed Embodiments

[0056] In order to clearly and completely describe the objectives, technical solutions of the present invention and make the advantages more clearly understood, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, not to limit the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0057] Embodiment 1, the present invention provides a technical solution: a method for constructing a large model data cleaning environment based on dynamic packaging, including the following steps:

[0058] Dynamically construct a basic operating environment package: For the target language and its version, generate a compressed file containing the language runtime, basic dependency libraries, and development packages through a package management tool, supporting multi-language and multi-version environment isolation; the basic operating environment package construction process includes a language version intelligent adaptation mechanism. By parsing the language version parameters in the task configuration, the corresponding package management tool is automatically called to generate the environment package, where: The Python environment uses conda to manage multi-version dependencies; the Java / Scala environment constructs a versioned dependency tree through Maven / SBT; after the environment package is generated, hash verification is used to ensure cross-node consistency.

[0059] Dynamically construct a business application package: Based on business requirements, package the cleaning logic code and its third-party dependency libraries into an independent compressed file, decoupled from the basic environment package; the business application package construction includes a dynamic dependency resolution and conflict detection mechanism, and the specific steps are: a) Scan the import statements of the cleaning script and the build configuration file; b) Generate a dependency relationship graph and detect version conflicts; c) Generate an application package containing precise version dependencies through virtual environment isolation technology; d) Support the injection of business customization parameters through the configuration file to achieve environment reuse.

[0060] Automated cluster environment deployment: Upload the two types of compressed packages to the cluster management node through the Spark client and dynamically bind them to the job running environment; the cluster environment deployment process adopts a two-layer distribution architecture, including: Client pre-verification: Submit a test job in Local mode to verify the integrity of the environment package and the correctness of the business logic; Cluster-level distribution: Use the --deploy-mode cluster parameter of SparkYarn to push the environment package to all worker nodes and specify the loading path through the spark.archives parameter; Runtime binding: Force the version to be specified through environment variables such as PYSPARK_PYTHON / JAVA_HOME, and combine JVM class loader isolation to achieve multi-environment parallelism.

[0061] Node-level environment dynamic loading and release: Load the specified environment package as needed in the working node's memory, automatically clean up resources after the job ends, and achieve coexistence of multiple versions of the environment and zero local dependencies; the dynamic loading and release mechanism includes the following technical features: a) Based on Spark's Application lifecycle management, each cleaning task runs as an independent Application, with an exclusive JVM instance and environment package memory space; b) Pre-load the environment package into the node memory through the distributed cache mechanism to reduce disk I / O overhead; c) Implement a resource recycling listener to automatically trigger the unloading of the environment package and release memory resources when the Application terminates; d) Support dynamically adjusting the environment package cache policy through configuration parameters to balance resource utilization and job startup speed.

[0062] Embodiment 2, based on Embodiment 1, proposes a construction system for the large model data cleaning environment based on dynamic packaging according to claim 5, including:

[0063] Client node: Used to dynamically construct a language version-specific basic running environment package and business function application package, and upload them to the cluster after local verification;

[0064] Cluster management node: Receive and store the uploaded environment packages and application packages, and be responsible for distributing the packages to the working nodes;

[0065] Working node: Dynamically load the specified environment package to execute the data cleaning task, and automatically clean up resources after the task ends;

[0066] The system supports isolation of multi-language and multi-version environments, and realizes zero-dependency installation of running nodes through the package binding mechanism to ensure environment consistency and compatibility.

[0067] 1) The Spark client node completes the dynamic packaging of the basic running environment, the construction of the application package, and the upload of these packages. According to the characteristics of the data cleaning environment, for example, some cleaning tasks require the selection of Python 3.10, Java 11.0, or Scala 2.12. The client node needs to package this specific running environment (including the language running environment and its dependent third-party libraries) into a compressed file, and at the same time package the specific application functions (including the dependent third-party packages) into another compressed file. These two types of compressed files are separated to maintain relative independent decoupling. Then, the Local method is used for Spark submit submission to test and verify whether the packaged data cleaning runs correctly. The correctness of the data cleaning task is verified through the local minimum resources and the test dataset. If the packaging is incorrect, then the packaging and Local test verification are repeated. Until the local test verification is completed, the Spark Yarn submission method will upload these local packages to the cluster management node at the same time. The following is the design of the core dynamic packaging class:

[0068] The IPackage interface specification defines two packaging methods, namely the basic running environment packaging doRuntimeEnvPackage() and the application packaging doApplicattionPackage().

[0069] PythonPackage implements the IPackage interface and uses the unique conda package management tool of Python to package the Python basic environment and the application, and at the same time selects the corresponding Python version environment package according to different version versions.

[0070] JavaPackage implements the IPackage interface and uses the maven package management tool to package the Java basic environment and the application, and at the same time selects the corresponding Python version environment package according to different version versions.

[0071] ScalaPackage implements the IPackage interface and uses the maven package management tool to package the Scala basic environment and the application, and at the same time selects the corresponding Python version environment package according to different version versions.

[0072] 2) The Spark cluster management node realizes the distribution of the basic running environment package and the application package to the Spark worker nodes. When submitting a data cleaning job using spark Yarn, by specifying the --deploy-mode cluster parameter, the Spark cluster management node will automatically distribute and push both the basic environment package and the application package to all worker nodes under the cluster.

[0073] 3) The Spark worker node realizes the dynamic binding and unbinding of the data cleaning environment running package on the current running node. When submitting through Spark submit, specify the running environment path of the current worker node (including the basic environment package and the application package) through spark.archives, and specify the specific running environment to be loaded in combination with PYSPARK_PYTHON. In this way, when a data cleaning job application starts running, Spark will load the necessary libraries and resources of the specified running environment into the memory of the worker node. In this way, each job submitted to the Spark cluster will run as an independent application, and each application will have its own execution environment (including JVM instances, class paths, libraries, and other resources). This can support data cleaning task nodes with only Python 2.7 version environment, or Python 3.10 version environment, or Java 8.0 version environment, or Scala 2.12 version environment, and can also support the worker node to have these language version environments running different data cleaning tasks simultaneously.

[0074] When the data cleaning job application is completed or terminated, Spark will automatically clean up the resources that are no longer used and release the memory space for other data cleaning job applications to use, which completes the dynamic unbinding of the data cleaning environment.

[0075] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a large model data cleaning environment based on dynamic packaging, characterized in that: It includes the following steps: Dynamically build the basic running environment package: For the target language and its version, generate a compressed file containing the language runtime, basic dependency libraries, and development packages through a package management tool, supporting multi-language and multi-version environment isolation; Dynamically build the business application package: Based on business requirements, package the cleaning logic code and its third-party dependency libraries into an independent compressed file, decoupled from the basic environment package; Automatically deploy the cluster environment: Upload the two types of compressed packages to the cluster management node through the Spark client and dynamically bind them to the job running environment; Node-level environment dynamic loading and release: On-demand load the specified environment package in the working node memory, and automatically clean up resources after the job ends, realizing the coexistence of multi-versions of the environment and zero local dependencies; 2. The method for constructing a large model data cleaning environment based on dynamic packaging according to claim 1, wherein: The basic running environment package construction process includes a language version intelligent adaptation mechanism. By parsing the language version parameters in the task configuration, automatically call the corresponding package management tool to generate the environment package. Among them: The Python environment uses conda to manage multi-version dependencies; The Java / Scala environment builds a versioned dependency tree through Maven / SBT; After the environment package is generated, hash verification is used to ensure cross-node consistency; 3. The method for constructing a large model data cleaning environment based on dynamic packaging according to claim 2, wherein: The business application package construction includes a dynamic dependency resolution and conflict detection mechanism. The specific steps are as follows: a) Scan the import statements of the cleaning script and the build configuration file; b) Generate a dependency graph and detect version conflicts; c) Generate an application package containing precise version dependencies through virtual environment isolation technology; d) Support injecting business customization parameters through a configuration file to achieve environment reuse; 4. A method for constructing a large model data cleaning environment based on dynamic packaging according to claim 3, characterized in that: The cluster environment deployment process adopts a two-layer distribution architecture, including: Client pre-verification: Submit a test job in Local mode to verify the integrity of the environment package and the correctness of the business logic; Cluster-level distribution: Use the --deploy-mode cluster parameter of Spark Yarn to push the environment package to all working nodes and specify the loading path through the spark.archives parameter; Runtime binding: Forcefully specify the version through environment variables such as PYSPARK_PYTHON / JAVA_HOME, and combine JVM class loader isolation to achieve multi-environment parallelism; 5. A method for constructing a large model data cleaning environment based on dynamic packaging according to claim 4, characterized in that: The dynamic loading and release mechanism includes the following technical features: a) Based on the Spark's Application lifecycle management, each cleaning task runs as an independent Application, with an exclusive JVM instance and environment package memory space; b) Pre-load the environment package into the node memory through the distributed cache mechanism to reduce disk I / O overhead; c) Implement a resource recovery listener to automatically trigger the unloading of the environment package and release memory resources when the Application terminates; d) Support dynamically adjusting the environment package cache policy through configuration parameters to balance resource utilization and job startup speed; 6. A construction system for the method of constructing a large model data cleaning environment based on dynamic packaging according to claim 5, characterized in that: It includes: Client node: Used to dynamically build the basic running environment package specific to the language version and the business function application package, and upload them to the cluster after local verification; Cluster management node: Receive and store the uploaded environment packages and application packages, and be responsible for distributing the packages to the working nodes; Worker Node: Dynamically load the specified environment package to execute the data cleaning task and automatically clean up resources after the task ends; The system supports multi-language and multi-version environment isolation, and realizes zero-dependency installation of running nodes through the package binding mechanism to ensure environment consistency and compatibility.

7. A construction system according to claim 6, characterized in that: The client node contains an intelligent packaging module, which specifically implements the following functions: a) Environment-aware construction: Parse the language and version parameters in the task configuration, and call the corresponding toolchain to generate the environment package. Conda is used for Python, and Maven is used for Java / Scala; b) Dependency conflict detection: Detect version conflicts by constructing a dependency relationship graph and generate an application package containing precise version dependencies; c) Localized pre-verification: Submit a test job through the Spark Local mode to verify the integrity of the environment package and the correctness of the business logic; d) Cluster deployment trigger: After verification, upload the two types of compressed packages to the cluster management node through the Spark Yarn mode.

8. A construction system according to claim 7, characterized in that: The cluster management node implements a two-layer environment distribution mechanism: Global storage layer: Use a distributed file system to store all version environment packages and establish a hash index to ensure cross-node consistency; Dynamic push layer: Trigger automatic distribution through the --deploy-mode cluster parameter, and push the specified environment package to the target worker node, supporting dynamic combination of the base package and the application package according to job requirements; The environment package distribution process includes a version verification mechanism to ensure that the environment loaded by the worker node is strictly consistent with the version built by the client.

9. A construction system according to claim 8, characterized in that: The worker node has an environment isolation runtime, which includes the following technical features: a) Memory-level loading: Specify the compressed package path through the spark.archives parameter, and force the loading of the specified version environment in combination with the PYSPARK_PYTHON / JAVA_HOME variable; b) JVM instance isolation: Each data cleaning job runs as an independent Application, with a dedicated JVM instance and class loader, supporting parallel multi-version environments; c) Automatic resource recycling: Register a job lifecycle listener to trigger the unloading of the environment package when the Application terminates and release memory resources; d) Cache optimization strategy: Support configuring the environment package cache threshold to balance resource utilization and job startup speed.

10. A construction system according to claim 9, characterized in that: The packaging module is designed using the strategy pattern and includes the following core components: IPackage interface: Define a unified packaging specification, including the doRuntimeEnvPackage() and doApplicationPackage() methods; Language-specific implementation classes: PythonPackage: Create a virtual environment through conda and generate a tar package containing the specified Python version and dependencies; JavaPackage: Build a zip package containing the specified JDK version and JAR dependencies through Maven; ScalaPackage: Generate a compressed package containing the Scala compiler and library dependencies through SBT; Each implementation class includes a version parser, which supports extracting language version parameters from the task configuration and mapping them to the corresponding basic environment template.

Images