Method and equipment for realizing data sandbox release based on automatic processing and storage medium

Abstract

The invention discloses a method and device for achieving data sandbox release based on automatic processing and a storage medium, and relates to the technical field of data processing.The method comprises the steps that a corresponding suite type is selected according to a release package type, and a corresponding suite service is called based on the suite type to construct a release package task; calling the suite service, generating a release package based on the release package task, and storing the release package; according to the storage result notice of the release package, a release approval process is initiated, and the release approval process is used for determining whether the release package is deployed to a target environment; and when the release approval process passes, deploying the release package to the target environment. According to the method, the publishing package task is constructed by selecting the suite type, the publishing package is generated and automatically stored, and the approval process is introduced, so that the standardability and traceability of publishing of the publishing package are ensured, the risk of human errors and environmental conflicts is reduced, and the deployment capability of the data sandbox is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device and storage medium for implementing data sandbox publishing based on automated processing. Background Art

[0002] Data sandboxes are usually used to solve the isolation problem between tasks in different environments of big data platforms and ensure the safe use of data between tasks. In big data platforms, current data sandbox technology usually creates development tasks based on online data to simulate real business scenarios. However, this method requires requesting data from different sources and explaining the data processing method to developers, who then process the data to meet the production requirements of online tasks.

[0003] Once the online requirements change, given the complexity of online environment tasks, developers are prone to omissions, mistakes, etc. in the task modification process. When the changed tasks are deployed to the online environment, production problems such as dirty data will occur, affecting the accuracy of data processing. Summary of the invention

[0004] The main purpose of this application is to provide a method, device and storage medium for data sandbox publishing based on automated processing, aiming to solve the technical problem that development task configuration errors are prone to occur when manually publishing and deploying services.

[0005] To achieve the above-mentioned purpose, an embodiment of the present application provides a method for publishing a data sandbox based on automated processing, and the method for publishing a data sandbox based on automated processing includes: Selecting a corresponding suite type according to the release package type, and calling a corresponding suite service based on the suite type to construct a release package task; Calling the suite service, generating a release package based on the release package task, and saving the release package; Initiate a release approval process according to the saving result notification of the release package, wherein the release approval process is used to determine whether to deploy the release package to the target environment; When the release approval process is passed, the release package is deployed to the target environment.

[0006] In one embodiment, before the steps of calling the suite service, generating a release package based on the release package task, and saving the release package, the steps include: According to the cluster information of the release package task, a mapping relationship is established between different environments according to the principle of the same type and the same name for the cluster information; Determine whether the mapping relationship is successfully established, and if the mapping relationship is successfully established, generate a plurality of the target environments.

[0007] In one embodiment, the step of determining whether the mapping relationship is successfully established, if the mapping relationship is successfully established, after the step of generating a plurality of the target environments, further comprises: Perform release detection on the release package task to obtain a release detection result; If the release detection result shows that the target environment does not meet the release conditions of the release package task, or the tasks or resources that the release package task depends on are not created in the target environment, it is impossible to generate a release package based on the release package task; When the release detection result passes, the release package is generated according to the release package task.

[0008] In one embodiment, the steps of calling the suite service, generating a release package based on the release package task, and saving the release package include: Calling the suite service to execute a command to save and publish the package; Based on the save publish package command, the suite service uses the task identifier of the publish package task to obtain the task or resource on which the publish package task depends; Based on the tasks or resources on which the release package task depends, the suite service generates a release package and mounts the release package to the sandbox shared disk.

[0009] In one embodiment, the release approval process is initiated according to the saving result notification of the release package, and the steps of the release approval process for determining whether to deploy the release package to the target environment include: Receiving a notification of a saving result of the publishing package; If the saving result notification is failed, the publishing task record status of the publishing package is updated to failure, and the publishing process of the publishing package is stopped; If the saving result notification is successful, the publishing approval process is initiated.

[0010] In one embodiment, when the release approval process is passed, the step of deploying the release package to the target environment includes: When the release approval process is passed, a release deployment command of the release package is sent to the suite service; Based on the release deployment command, the suite service obtains the release package from the sandbox shared disk according to the release package identifier of the release package; The suite service parses the release package, obtains resources of the release package task, and uploads the resources to the target environment.

[0011] In one embodiment, after the steps of parsing the release package, obtaining resources of the release package task, and uploading the resources to the target environment, the process further includes: The suite service executes a creation, update or deletion operation of the release package task in the target environment according to the task change type of the release package; When the suite service completes the creation, update or deletion operation of the release package task, the deployment of the release package in the target environment is completed.

[0012] In one embodiment, after the suite service completes the creation, update or deletion operation of the release package task and the step of deploying the release package in the target environment, the process further includes: The publishing result of the publishing package is obtained, and based on the publishing result, the publishing status of the publishing package task is updated.

[0013] An embodiment of the present application also provides a device for publishing data sandboxes based on automated processing, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the method for publishing data sandboxes based on automated processing as described above.

[0014] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data sandbox publishing method based on automated processing as described above are implemented.

[0015] The embodiment of the present application discloses a method for implementing data sandbox publishing based on automated processing, by selecting a corresponding suite type according to the release package type, and calling the corresponding suite service based on the suite type to build a release package task; calling the suite service, generating a release package based on the release package task, and saving the release package; initiating a release approval process based on the save result notification of the release package, the release approval process is used to determine whether to deploy the release package to the target environment; when the release approval process passes, the release package is deployed to the target environment. The present application ensures the standardization and traceability of the release package release by selecting the suite type to build the release package task, generating and automatically saving the release package, and introducing the approval process, reducing the risk of human error and environmental conflicts, and improving the deployment capability of the data sandbox. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a first embodiment of a method for implementing data sandbox publishing based on automated processing according to an embodiment of the present application; Figure 2 This is a flow chart of a second embodiment of a method for implementing data sandbox publishing based on automated processing according to an embodiment of the present application; Figure 3 This is a flow chart of a third embodiment of a method for implementing data sandbox publishing based on automated processing according to an embodiment of the present application; Figure 4 This is a flow chart of a fourth embodiment of a method for implementing data sandbox publishing based on automated processing according to an embodiment of the present application; Figure 5 This is a flow chart of a fifth embodiment of a method for implementing data sandbox publishing based on automated processing according to an embodiment of the present application; Figure 6 A brief flowchart of a method for implementing data sandbox publishing based on automated processing according to an embodiment of the present application; Figure 7 This is a schematic diagram of the structure of a data sandbox publishing device based on automated processing involved in an embodiment of the present application.

[0017] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0018] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] Data sandboxes are usually used to solve the isolation problem between tasks in different environments of big data platforms and ensure the safe use of data between tasks. In big data platforms, current data sandbox technology usually creates development tasks based on online data to simulate real business scenarios. However, this method requires requesting data from different sources and explaining the data processing method to developers, who then process the data to meet the production requirements of online tasks.

[0020] Once the online requirements change, given the complexity of online environment tasks, developers are prone to omissions, mistakes, etc. in the task modification process. When the changed tasks are deployed to the online environment, production problems such as dirty data will occur, affecting the accuracy of data processing.

[0021] In order to solve the above-mentioned defects existing in the related art, the embodiment of the present application proposes a method for implementing data sandbox publishing based on automated processing, which selects the corresponding suite type according to the release package type, and calls the corresponding suite service based on the suite type to build a release package task; calls the suite service, generates a release package based on the release package task, and saves the release package; initiates a release approval process based on the saving result notification of the release package, and the release approval process is used to determine whether to deploy the release package to the target environment; when the release approval process passes, the release package is deployed to the target environment. This application ensures the standardization and traceability of the release package by selecting the suite type to build the release package task, generating and automatically saving the release package, and introducing the approval process, reducing the risk of human errors and environmental conflicts, and improving the deployment capability of the data sandbox.

[0022] It should be noted that the execution subject of this embodiment can be a data sandbox, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a data sandbox publishing device based on automated processing that can achieve the above functions. The following takes the data sandbox as an example to illustrate this embodiment and the following embodiments.

[0023] The first embodiment of the present application proposes a method for implementing data sandbox publishing based on automated processing. Please refer to Figure 1 The method comprises steps S10 to S40: Step S10: selecting a corresponding suite type according to the release package type, and calling a corresponding suite service based on the suite type to construct a release package task.

[0024] During the deployment of the data sandbox, the release package usually contains application code, configuration files, dependent libraries and other resources, and different applications may require different deployment logic and environment configuration. Therefore, in order to ensure that the release package can run correctly in the target data sandbox environment, you first need to determine which suite type to use based on the type of release package.

[0025] Suite types target big data capabilities, such as data development and data integration. For example, an application for data development requires a suite that includes a data processing framework, a computing engine, and a storage service. Different suite types are designed to meet different application and release package tasks, including service resources required at various stages from development, testing to deployment, to ensure that the release package can be deployed and run smoothly in the target environment. The role of the data sandbox is to deploy the capabilities corresponding to these suite types to different target environments and to achieve isolation between target environments to ensure data security and independence.

[0026] In this embodiment, the sandbox service, as the core service component of the data sandbox publishing process, is adapted to multiple task types and can select and build different publishing package types based on different suite types, such as data development tasks, data integration synchronization types, etc. The sandbox service associates the publishing package type with the suite type by storing a pre-set mapping relationship, so as to find the corresponding suite type according to the input publishing package type.

[0027] After selecting the release package type, call the corresponding suite service according to the suite type corresponding to the release package type to obtain the relevant parameters required for the release package task. For example, for a data development task, you can obtain information such as functions, resources, and parameters associated with the data development task, and you can also count the number of tasks associated with the data development task.

[0028] When calling the suite service, the sandbox service will pass the required context information to the suite service to help the suite service more accurately obtain the resources and parameters required for the release package task. At the same time, after receiving the call request, the suite service will use its own logic and algorithms to provide corresponding support for building the release package task based on the received information (such as the release package type, task details, etc.), ensuring that the built release package task can be smoothly executed in the release process. For example, the suite service can call different internal modules or external interfaces to obtain and process the required data, such as calling the storage service to obtain the required code files or configuration files, and calling the resource management service to obtain resources such as dependent libraries.

[0029] It should be noted that the suite service is the execution unit in the corresponding suite type, which is responsible for executing specific operations and converting the various service functions provided in the suite type into an operable process.

[0030] In an optional implementation, this embodiment supports batch selection of release package tasks and obtains the task change type corresponding to the release package tasks. When there are multiple release package tasks to be processed, batch selection can complete certain operations of multiple release package tasks at one time, avoiding the need for users to repeat the same process one by one.

[0031] It should be noted that the task change type is determined according to the status of the release package task at different stages and different operations, including creation (or addition), update or deletion, which is used to guide the specific operation type of the release package task in the target environment. Among them, the release package task that has not been released to the target environment belongs to the addition type task; the release package task that has been edited after release belongs to the update type task; the release package task that has been deleted in the current environment belongs to the deletion type task.

[0032] The sandbox service can create multiple types of release package tasks according to different change types, which improves the flexibility and adaptability of the sandbox service. At the same time, the sandbox service also supports visual operations, allowing users to view the parameter information associated with the corresponding release package task, including the basic information of the release package task, dependent resources, task status, and change type. Through the visual interface, users can understand the detailed information of the release package task, discover possible problems in a timely manner, and adjust and operate the release package task, thereby improving the success rate of the release package task to the target environment.

[0033] For example, the user can view which parameters are modified by the update type task or view the detailed information of the newly added type task through the visual interface to confirm whether the release conditions are met. By viewing and adjusting the detailed information of the release package task, errors can be avoided in the subsequent release process, thereby improving the success rate of the release package deployment to the target environment.

[0034] Step S20: calling the suite service, generating a release package based on the release package task, and saving the release package.

[0035] In this embodiment, after creating a release package task, the sandbox service will call the suite service to execute the instruction to save the release package. Specifically, the constructed release package task is used as input, and the suite service integrates and encapsulates the information in the release package task according to its own algorithm and logic. For example, the suite service collects various resources required for the release package task (such as code files, configuration files, data files, etc.), and organizes and stores this information in a specific format (such as JSON format, XML format or custom binary format) to form a complete release package.

[0036] While calling the suite service to generate a release package, the sandbox service saves the snapshot data of the release package task selected by the user on the page to the sandbox database to persist the release record of the release package task, so that subsequent users can view the release progress and detailed information of the release package task.

[0037] When the suite service collects various resources required for the release package task, it obtains the detailed information of the selected release package task and the resources that the release package task depends on, including reading corresponding files or data from different storage locations, or compressing or encrypting data, according to the task identifier of the release package task passed by the sandbox service. Then, these resources are packaged into a release package to ensure the integrity and security of the release package.

[0038] For example, the node of the offline development task includes the dependent resource jar packages, udf functions, etc. After the suite service downloads the resources that the offline development task depends on, it saves them in the release package corresponding to the offline development task.

[0039] For the generated release package, save the release package to a specific storage location. The storage location includes the sandbox shared disk, local disk, network storage device, such as NAS (Network Attached Storage) storage, cloud storage service, etc. The save operation can be implemented by calling the storage system interface or using the file system operation function. At the same time, the storage operation is monitored to ensure that the release package is saved successfully. If the save fails, the corresponding error handling will be performed, such as recording the error log, retrying the operation, notifying the administrator, or updating the release task record status of the release package.

[0040] When storing release packages, store them according to certain naming rules. You can use the release package task name, timestamp, task identifier and other information to form a file name, which is convenient for finding and managing the subsequent release process. For example, after creating the release package corresponding to the release package task, use the naming rule of the release package task's task identifier + timestamp to save the release package and mount it to the sandbox shared disk, so that the sandbox service and the suite service can share the data and files in the release package.

[0041] In an optional implementation scheme, for the release packages that have been successfully saved in the sandbox shared disk, a scheduled task can be set, such as every six months or every month, to regularly clean up the saved release packages in the sandbox shared disk to reduce the storage space occupied.

[0042] Step S30: Initiate a release approval process based on the saving result notification of the release package, wherein the release approval process is used to determine whether to deploy the release package to the target environment.

[0043] In this embodiment, after the release package is successfully saved, the release and deployment of the data sandbox needs to comply with the release requirements such as specifications, security and resource management, to avoid the risks that may be caused by the unaudited release package being randomly deployed in the target environment, such as security vulnerabilities, resource conflicts or inconsistencies with business logic. Therefore, through the release approval process, human errors can be reduced and the accuracy of the release package deployment can be ensured.

[0044] The save result notification is an information notification generated after the suite service completes the save operation of the release package in step S20, including information such as whether the save operation is successful, the save location, and the save time. Then, the save result notification will be sent to the sandbox service, which will serve as the basis for the sandbox service to initiate the release approval process.

[0045] In this embodiment, when the sandbox service receives a saving result notification indicating that the publishing package fails to be saved, the publishing process is not continued, and the publishing task record status of the publishing package in the sandbox database is updated to failure.

[0046] When the sandbox service receives the notification of the successful saving of the release package, it will automatically initiate the release approval process. After receiving the release package release process, the process approver can perform the approval operation in the process agent to decide whether to pass the release package release process.

[0047] If the release approval process is rejected, the release package will not be allowed to be released to the target environment, and the release process ends.

[0048] If the release approval process is passed, the release process will continue and enter the next step of deployment operation. Specifically, the deployment process of the release package will be executed, the release package parameters will be assembled, and the suite service will be called to execute the task release method of the release package.

[0049] This embodiment can reduce the risk of human errors and violations by initiating a release approval process and ensure visual tracking of the release process.

[0050] Step S40: When the release approval process is passed, the release package is deployed to the target environment.

[0051] In this embodiment, when the release package is evaluated and reviewed and is considered to meet the deployment conditions, the release package is deployed to the target environment to complete the data sandbox release operation.

[0052] The target environment refers to the environment where the release package is ultimately deployed. The target environment can be a physical server cluster, a virtual data center, a cloud computing service environment, or an isolated data sandbox environment. The target environment is the environment where the release package ultimately runs.

[0053] When the release approval process is passed, the sandbox service will find the release package to be deployed based on the stored release package information, including its storage location (such as the sandbox shared disk, cloud storage service, etc.) and the release package identifier of the release package.

[0054] Next, the sandbox service calls the corresponding suite service to extract the information in the release package, for example, deploying the code, configuration files, dependent libraries and other resources in the release package to the target environment.

[0055] During the deployment process, you first need to complete resource allocation and environment configuration operations for the target environment based on the resources and other data in the release package. Then, according to the release package tasks in the release package, you need to allocate the required computing resources for the tasks, such as CPU (Central Processing Unit), memory, storage, etc., and create the required operating environment, such as installing the corresponding software environment and starting services.

[0056] Different types of release packages may use different deployment methods. For example, for containerized release packages, use container orchestration tools to deploy containers to nodes in the target environment. For traditional application release packages, use automated deployment scripts to install applications on servers.

[0057] In addition, during the deployment process, the deployment operation will be monitored to ensure the success of each operation. If a deployment error occurs, the error information will be recorded for subsequent troubleshooting and resolution. At the same time, the deployment progress can also be updated and displayed so that users can understand the deployment status of the release package.

[0058] This embodiment reduces the complexity of manual configuration and the risk of configuration errors in development tasks by automatically selecting suite types and calling suite services. It also implements strict review of release packages through the release approval process, ensuring the quality of release packages and visual tracking of the release process.

[0059] Based on the above embodiments, please refer to Figure 2 The second embodiment of the present application proposes a method for implementing data sandbox publishing based on automated processing, and before step S20, includes steps S201 to S202: Step S201: According to the cluster information of the release package task, a mapping relationship is established between different environments based on the principle of the same type and the same name for the cluster information.

[0060] Step S202: Determine whether the mapping relationship is successfully established. If the mapping relationship is successfully established, generate a plurality of the target environments.

[0061] In the data sandbox environment, there are multiple clusters, and different clusters contain different resources and services and need to be deployed in different environments. In order to ensure the accurate deployment and resource allocation of release package tasks in different environments, it is necessary to establish a reasonable mapping relationship between different environments to better manage and schedule resources.

[0062] It should be noted that cluster information includes various information about clusters related to the release package task, such as the type of cluster, the name of the cluster, and the resource information owned by the cluster, such as the number of servers, storage capacity, network configuration, etc. The mapping relationship refers to an association established between different environments based on the principle of the same type and the same name. The same type refers to clusters with similar technical architectures and functional characteristics, and the same name refers to clusters with the same name. By establishing a mapping relationship between different environments based on this principle, similar clusters in different environments can be better managed and operated in a unified manner, ensuring the consistency of release package tasks between different environments.

[0063] In this embodiment, cluster information of the release package task is obtained, such as cluster data source, computing resource queue mapping, parameters, etc. According to the principle of the same type and the same name, a mapping relationship is automatically established between different environments. This process matches and maps the resources and configurations in the source environment with the corresponding resources in the target environment by analyzing the structure and attributes of the cluster information.

[0064] For example, there is a data source named "data_source_1" in the source environment. Then, by searching for a data source with the same name in the target environment, a mapping relationship can be established between the data source with the same name in the target environment and the data source "data_source_1" in the source environment.

[0065] If the mapping relationship is successfully established, multiple target environments are generated. These target environments are candidate environments for subsequent deployment of the release package.

[0066] It should be noted that generating multiple target environments is to provide the option of switching target environments, and it can also support multi-environment deployment strategies, such as gradually verifying and deploying release packages in development, test, and production environments.

[0067] If the mapping relationship is not successfully established, the subsequent release detection phase cannot be carried out, and an error message is returned to prompt the user to check and adjust.

[0068] Furthermore, step S202 further includes steps S2021 to S2023: Step S2021: Perform release detection on the release package task to obtain a release detection result.

[0069] In this embodiment, a release test is performed on the release package task to verify whether the target environment meets the release conditions. The release test includes release environment test and task test, etc. The result of the release test will determine whether to continue the subsequent release process.

[0070] Release testing is a key step to ensure that the release package task can run smoothly in the target environment. Release testing results are generated by checking the configuration, resource availability, and task dependencies of the target environment.

[0071] The release detection process involves checking key elements of the target environment, such as database tables, computing resources, and network configuration, to ensure that the above information is consistent with the release requirements of the release package task.

[0072] For example, if the release package task depends on specific database tables or computing resource queues, release detection verifies whether these resources have been created and are available in the target environment.

[0073] Step S2022: If the release detection result shows that the target environment does not meet the release conditions of the release package task, or the tasks or resources on which the release package task depends are not created in the target environment, a release package cannot be generated based on the release package task.

[0074] In this embodiment, it is determined whether the target environment meets the release condition or whether the task or resource on which the release package task depends is created in the target environment according to the release detection result.

[0075] If the release detection result shows that the target environment does not meet the release conditions of the release package task, or the tasks or resources that the release package task depends on are not created in the target environment, the release process is terminated and an error message is returned.

[0076] The release detection results of this embodiment ensure that the release package will be generated and deployed only when the target environment is fully prepared. For example, if a key dependent resource is missing in the target environment, or a necessary configuration parameter is not set correctly, the sandbox service cannot generate a release package. This approach can avoid release failures or runtime errors caused by environment mismatch.

[0077] It should be noted that the release conditions refer to whether the hardware resources of the target environment meet the requirements of the release package task and whether the required software environment matches the release package task. The release detection process verifies whether these release conditions are met, thereby deciding whether to continue the subsequent release package creation process.

[0078] Step S2023: When the release detection result passes, the release package is generated according to the release package task.

[0079] In this embodiment, when the release test result is passed, the sandbox service will generate a release package according to the release package task. In the process of generating the release package, the sandbox service will call the suite service to integrate and package the task code, configuration file, dependent resources, etc. required for the release package task to form a release package.

[0080] The generated release package contains all the necessary information, tasks, and resources required for task deployment to ensure that the release package task can run independently in the target environment. For example, the suite service generates a release package folder in the form of a JSON file, or generates a zip package of the release package, including the task code, dependent JAR packages, and configuration files required by the release package task, so that it can be quickly verified and loaded during deployment.

[0081] Based on the above embodiments, please refer to Figure 3 In the third embodiment of the present application, the data sandbox publishing method based on automated processing is implemented, and step S20 may further include steps S210 to S230: Step S210: calling the suite service to execute a command to save and publish the package.

[0082] In the data sandbox publishing process, the generation and storage of the release package is the key link to ensure that the release package task can be successfully deployed to the target environment. As a virtual environment that isolates and manages the data analysis environment, the data sandbox needs to use automated means to ensure that the dependent resources of the release package task are correctly identified, packaged, and stored in a shared location for subsequent deployment operations.

[0083] Therefore, when the release package task passes the task detection, the sandbox service will call the suite service to execute the command to save the release package. The suite service is the core component for generating and storing release packages. By receiving the command to save the release package, the suite service will start the packaging and storage process for the release package task.

[0084] Step S220: Based on the save publish package command, the suite service uses the task identifier of the publish package task to obtain the task or resource on which the publish package task depends.

[0085] Step S230: Based on the tasks or resources on which the release package task depends, the suite service generates a release package and mounts the release package to the sandbox shared disk.

[0086] According to the task identifier of the release package task in the save release package command, the suite service can obtain the release package task details and the tasks or resources that the release package task depends on, such as data sources, configuration files, and dependent libraries, in real time. Then, based on the acquired data such as the tasks or resources that the release package task depends on, the suite service packages them to generate a release package. At the same time, the release package is mounted to the sandbox shared disk.

[0087] Creating a release package involves packaging the resources that the task depends on, such as JAR packages and UDF functions, into a deployable format, such as a folder or ZIP package, and naming it with a release package identifier and timestamp to ensure version control and tracking of the release package.

[0088] It should be noted that the "Sandbox Shared Disk" is a shared storage area that can be used by sandbox services and suite services, etc., to facilitate the sharing of data and files between sandbox services and suite services. The generated release package is stored in the sandbox shared disk, which facilitates the subsequent sandbox services and suite services to access and use the release package when deploying the release package task.

[0089] Based on the above embodiments, please refer to Figure 4 The fourth embodiment of the present application proposes a method for implementing data sandbox publishing based on automated processing, wherein step S30 includes steps S310 to S330: Step S310: receiving a notification of a saving result of the release package.

[0090] In this embodiment, after the package service completes the save operation of the release package, it will generate a save result notification, which includes information such as whether the save operation is successful, the save location, and the save time, etc. The package service will send the save result notification to the sandbox service.

[0091] Step S320: If the saving result notification is failure, the publishing task record status of the publishing package is updated to failure, and the publishing process of the publishing package is stopped.

[0092] In this embodiment, after the suite service sends the notification information of the failed save to the sandbox service, the sandbox service will update the release task record status of the release package to failure and stop the release process of the release package to avoid deployment failure and other problems that may be caused by further operations.

[0093] Step S330: If the saving result notification is successful, the publishing approval process is initiated.

[0094] In this embodiment, if the save result notification sent by the suite service to the sandbox service is successful, it indicates that the release package has been successfully saved and subsequent operations can continue. At this time, the sandbox service will automatically initiate the release approval process, pass the detailed information of the release package to the approval process, and notify the relevant approvers or automated approval mechanisms to evaluate whether the current release package can be deployed to the target environment.

[0095] Based on the above embodiments, please refer to Figure 5 The fifth embodiment of the present application proposes a method for implementing data sandbox publishing based on automated processing, wherein step S40 includes steps S410 to S430: Step S410: After the release approval process is passed, a release deployment command of the release package is sent to the suite service.

[0096] When the release approval process is passed, the sandbox service sends the release package deployment command to the suite service. The release package deployment command is used to ensure that only approved release packages enter the deployment phase when notifying the suite service to start the release package deployment operation.

[0097] Publishing deployment commands is usually delivered through API calls or message queues.

[0098] Step S420: Based on the release deployment command, the suite service obtains the release package from the sandbox shared disk according to the release package identifier of the release package.

[0099] After receiving the release deployment command of the release package, the suite service searches for and obtains the release package corresponding to the command in the sandbox shared disk according to the release package identifier of the release package as an index.

[0100] Step S430: the suite service parses the release package, obtains resources of the release package task, and uploads the resources to the target environment.

[0101] In this embodiment, the suite service parses the obtained release package, reads the metadata and resource information in the release package, and obtains the task details of the release package task in the release package and the resources associated with the task. These resources include code files, configuration files, dependent libraries, etc. Then, the suite service uploads these resources to the storage location or computing node in the target environment by calling the resource management service or network transmission service of the target environment.

[0102] Exemplarily, the resource jar package, reference parameters, etc. that the task depends on are obtained in the offline development task node, the resources associated with the task are uploaded in advance, and the dependent parameter information is created in the target environment.

[0103] Furthermore, after step S430, steps S440 to S450 are also included: Step S440: the suite service executes a creation, update or deletion operation of the release package task in the target environment according to the task change type of the release package.

[0104] After configuring the target environment parameters, the suite service creates, updates, or deletes the release package task according to the task change type of the release package task in the release package, and publishes the corresponding release package task to the target environment. For example, if the offline development suite publishes a new release package task, it will automatically replace the data and environment variables in the associated task parameters to the target environment, thereby achieving data and computing isolation between development mode tasks and online mode tasks.

[0105] It should be noted that the associated task parameters refer to the configuration information and parameter settings related to the published package task. These parameters are used to control the behavior and resource references of the release package task in different environments (such as development environment, test environment, and production environment).

[0106] For example, in the data sandbox, a developer writes a SQL task that needs to reference specific data tables and computing resources. In the development environment, the task references a table in the test database; in the production environment, the task references a table in the formal database. By using associated task parameters, the suite service can automatically replace these parameters in different environments to ensure that the task can run correctly.

[0107] In the release process, when the release package task is deployed to the target environment, the sandbox service calls the suite service to automatically replace the associated task parameters according to the configuration of the target environment.

[0108] Step S450: After the suite service completes the creation, update or deletion operation of the release package task, the deployment of the release package in the target environment is completed.

[0109] When the package service completes the creation, update, or deletion of the release package task, it indicates that the deployment of the release package task in the target environment has been completed.

[0110] Furthermore, after step S450, step S460 is also included: Step S460: Obtain the publishing result of the publishing package, and based on the publishing result, update the publishing status of the publishing package task.

[0111] After the suite service completes the deployment of the release package task, it will send the result of the release package execution to the sandbox service. After the sandbox service obtains the release result of the release package by checking the return value of the deployment operation, the running status report of the target environment, or other monitoring mechanisms, it will update the release status of the release package task according to the release result. For example, after the suite service successfully releases the release package task, the sandbox service detects the release result callback sent by the suite service, and will update the release status of the release package task from "deploying" to "deployed", or update the release status from "deploying" to "deployment failed".

[0112] In addition, the sandbox service also obtains the release operation record of the suite service when publishing the release package task, including the operation process and the reasons for success or failure, to facilitate subsequent task management and problem troubleshooting.

[0113] This embodiment realizes the automation of publishing package tasks to the target environment through the sandbox service, and the sandbox service is adapted to multiple suite services, and centrally and uniformly manages the publishing package tasks, so that the suite service only needs one set of code to switch between different environments, avoiding the modification of the suite service code and data, and ensuring the efficiency of suite service development.

[0114] Based on the above embodiments, the method for implementing data sandbox publishing based on automated processing in the sixth embodiment proposed by the present application further includes: In this embodiment, the sandbox service adopts a container-based publishing model to achieve multi-level user data and operation isolation for different users, different projects, and different environments.

[0115] Specifically, the sandbox service uses the sandbox shared library as a unified database for data sharing. When each user creates a release package task, a permission check is performed when the user accesses the sandbox shared library based on the user's identity, current project, and environment. For example, when a user attempts to read or write a release package from the sandbox shared library, the user's permission information, such as user ID, project ID, target environment ID, and corresponding operation permissions, is used to determine whether the user has the corresponding operation permissions. By intercepting user requests and verifying permissions, it is ensured that users can only access and operate the release package data for which they have permissions.

[0116] In addition, when performing a release package save operation, the release package is marked, and the mark contains information such as user ID, project ID, and target environment ID. When a user initiates a release package acquisition request, the query statement or operation request is modified and filtered according to the user's current target environment and permissions. For example, for SQL query statements, the user ID, project ID, and target environment ID filter conditions are automatically added to ensure that users can only obtain and operate release package data under their target environment and project. For release package data in different environments, the corresponding release package identifier is added during storage and query to avoid confusion between release package data in different environments.

[0117] When a user switches between different projects or target environments, the user's session information is updated according to the user's new target environment and project information, including the current user, project, and target environment identifiers. After switching the target environment, through user permissions and release package data filtering, the user can only see and operate the release package data of the corresponding target environment for which he has permissions based on the updated session information.

[0118] This application ensures the security of different suite services and release package data through a multi-level control method. It does not need to deploy a cluster environment separately for each user, reducing the cost of environment deployment, while also ensuring that data between different target environments is not contaminated.

[0119] For example, in order to help understand the implementation process of the data sandbox publishing method based on automated processing obtained by combining this embodiment with the above embodiments, please refer to Figure 6 , Figure 6 A brief flow chart of a method for implementing data sandbox publishing based on automated processing is provided, specifically: The user starts to select the release package task through the interface. After the sandbox service receives the user's release package task selection request, it obtains the task details of the release package task and the resource information required for the release package task in real time, and completes the parameter assembly according to the required resource information. Then, based on the release package task selected by the user, the release strategy mapping is executed. Through the relevant cluster information of the release package task configured by the user, for example, a release package task is configured with cluster data source, computing resource queue mapping, parameters, etc., it will automatically map according to the principle of the same type and the same name, establish a mapping relationship between different environments, and obtain the target environment corresponding to multiple release package tasks, so that the release package task can select the target environment for release.

[0120] Only when the release strategy mapping is passed, the release package task will be subject to the next release test, including release environment test and task test. Determine whether the target environment corresponding to the release package task meets the release conditions of the release package task, such as whether the corresponding database table of the data integration synchronization task exists in the target environment. Next, check whether the tasks and resources that the release package task depends on have been created in the target environment. For release package tasks that fail the release test, the release package cannot be created. Only after passing the release test, that is, pre-checking the release package deployment in advance to ensure the quality of task release during the release process, can the sandbox service continue to create the release package.

[0121] The sandbox service first saves the snapshot data of the release package task to the sandbox database to persist the release record of the release package, so that subsequent users can view the task release progress and details. Then, the release package is created by calling the suite service. Specifically, the suite service obtains the task details of the release package task in real time according to the sandbox service's save release package instruction, then extracts the tasks and dependent resources of the release package task, and packages the extracted data to generate a release package, and mounts the release package to the sandbox shared disk.

[0122] After creating a release package, the suite service will send a notification of saving the release package result to the sandbox service. After the sandbox service monitors the release package save callback message, it will determine whether the release package save operation is successful. When the release package is saved successfully, the sandbox service will submit the workflow, that is, the release approval process. Workflow approval is used to verify whether the release package meets the preset release conditions, such as environmental compatibility, resource availability, etc. Among them, the approval process includes manual review or automated inspection.

[0123] When the sandbox service receives the workflow result callback notification and determines that the release process has been approved, the sandbox service will send the release package deployment command to the suite service. After receiving the release package deployment command, the suite service obtains the corresponding release package file from the shared storage according to the release package identifier, and parses the release package file to obtain the release dependency resources and parameters of the release package task. At the same time, it also obtains and parses the task record of the release package task.

[0124] Subsequently, the suite service uploads the resources in the release package to the target environment, including the creation of database tables, uploading of resource files, setting of environment variables, etc. The suite service submits task creation operations of different task change types in the target environment according to the task change type (addition, update, or deletion) in the release package.

[0125] After completing resource upload and environment configuration, the suite service will execute the release package task, such as starting the task, running the job flow, etc. After the release package task is executed, the suite service will feedback the execution result of the release package to the sandbox service. The sandbox service monitors the callback message of the release package execution, and after determining that the release package task is successfully executed, it will synchronously update the release status of the release package task, completing the entire data sandbox release automation process.

[0126] This embodiment realizes the automation of the data sandbox publishing process, reduces manual intervention, and improves publishing efficiency and accuracy.

[0127] An embodiment of the present application provides a device for publishing data sandboxes based on automated processing, and the device for publishing data sandboxes based on automated processing includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for publishing data sandboxes based on automated processing in the above-mentioned embodiment one.

[0128] Reference below Figure 7 , which shows a schematic diagram of the structure of a data sandbox publishing device based on automated processing suitable for implementing the embodiment of the present application. The data sandbox publishing device based on automated processing in the embodiment of the present application may include various hardware and software components for implementing a data sandbox publishing method based on automated processing. Figure 7 The data sandbox publishing device based on automated processing is shown as merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0129] like Figure 7As shown, the data sandbox publishing device based on automated processing may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. In the random access memory 1004, various programs and data required for the operation of the data sandbox publishing device based on automated processing are also stored. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data sandbox publishing device based on automated processing to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a data sandbox publishing device based on automated processing with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0130] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0131] The data sandbox publishing device based on automated processing provided by the present application adopts the data sandbox publishing method based on automated processing in the above embodiment, which can solve the technical problem that development task configuration errors are prone to occur when manually publishing and deploying services. Compared with the prior art, the beneficial effects of the data sandbox publishing device based on automated processing provided by the present application are the same as the beneficial effects of the data sandbox publishing method based on automated processing provided by the above embodiment, and the other technical features of the data sandbox publishing device based on automated processing are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0132] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0133] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0134] An embodiment of the present application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, and the computer-readable program instructions are used to execute the data sandbox publishing method based on automated processing in the above-mentioned embodiment.

[0135] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM, CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequencies (RF, Radio Frequency), etc., or any suitable combination of the above.

[0136] The above-mentioned computer-readable storage medium may be included in the data sandbox publishing device based on automated processing; or it may exist independently without being assembled into the data sandbox publishing device based on automated processing.

[0137] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the data sandbox publishing device based on automated processing, the data sandbox publishing device based on automated processing: selects the corresponding suite type according to the release package type, and calls the corresponding suite service based on the suite type to build a release package task; calls the suite service, generates a release package based on the release package task, and saves the release package; initiates a release approval process based on the saving result notification of the release package, and the release approval process is used to determine whether to deploy the release package to the target environment; when the release approval process passes, the release package is deployed to the target environment.

[0138] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] The modules involved in the embodiments described in the present application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0141] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data sandbox publishing method based on automated processing, and can solve the technical problem that development task configuration errors are prone to occur when manually publishing and deploying services. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the data sandbox publishing method based on automated processing provided by the above-mentioned embodiment, and will not be repeated here.

[0142] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the data sandbox publishing method based on automated processing as described above.

[0143] The computer program product provided by the present application can solve the technical problem that development task configuration errors are prone to occur when manually publishing and deploying services. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as the beneficial effects of the data sandbox publishing method based on automated processing provided by the above embodiment, which will not be repeated here.

[0144] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

[0145] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0147] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for implementing data sandbox publishing based on automated processing, characterized in that: The method for implementing data sandbox publishing based on automated processing includes: Selecting a corresponding suite type according to the release package type, and calling a corresponding suite service based on the suite type to construct a release package task; Calling the suite service, generating a release package based on the release package task, and saving the release package; Initiate a release approval process according to the saving result notification of the release package, wherein the release approval process is used to determine whether to deploy the release package to the target environment; When the release approval process is passed, the release package is deployed to the target environment.

2. The method for implementing data sandbox publishing based on automated processing according to claim 1, characterized in that: Before the step of calling the suite service, generating a release package based on the release package task, and saving the release package, the method includes: According to the cluster information of the release package task, a mapping relationship is established between different environments according to the principle of the same type and the same name for the cluster information; Determine whether the mapping relationship is successfully established, and if the mapping relationship is successfully established, generate a plurality of the target environments.

3. The method for implementing data sandbox publishing based on automated processing as claimed in claim 2, characterized in that: The step of determining whether the mapping relationship is successfully established and, if the mapping relationship is successfully established, generating a plurality of target environments, further includes: Perform release detection on the release package task to obtain a release detection result; If the release detection result shows that the target environment does not meet the release conditions of the release package task, or the tasks or resources that the release package task depends on are not created in the target environment, it is impossible to generate a release package based on the release package task; When the release detection result passes, the release package is generated according to the release package task.

4. The method for implementing data sandbox publishing based on automated processing according to claim 1, characterized in that: The steps of calling the suite service, generating a release package based on the release package task, and saving the release package include: Calling the suite service to execute a command to save and publish the package; Based on the save publish package command, the suite service uses the task identifier of the publish package task to obtain the task or resource on which the publish package task depends; Based on the tasks or resources on which the release package task depends, the suite service generates a release package and mounts the release package to the sandbox shared disk.

5. The method for implementing data sandbox publishing based on automated processing according to claim 1, characterized in that: The step of initiating a release approval process according to the notification of the saving result of the release package, wherein the release approval process is used to determine whether to deploy the release package to the target environment includes: Receiving a notification of a saving result of the publishing package; If the saving result notification is failed, the publishing task record status of the publishing package is updated to failure, and the publishing process of the publishing package is stopped; If the saving result notification is successful, the publishing approval process is initiated.

6. The method for implementing data sandbox publishing based on automated processing according to claim 1, characterized in that: When the release approval process is passed, the step of deploying the release package to the target environment includes: When the release approval process is passed, a release deployment command of the release package is sent to the suite service; Based on the release deployment command, the suite service obtains the release package from the sandbox shared disk according to the release package identifier of the release package; The suite service parses the release package, obtains resources of the release package task, and uploads the resources to the target environment.

7. The method for implementing data sandbox publishing based on automated processing according to claim 6, characterized in that: After the steps of parsing the release package, obtaining resources of the release package task, and uploading the resources to the target environment, the method further includes: The suite service executes a creation, update or deletion operation of the release package task in the target environment according to the task change type of the release package; When the suite service completes the creation, update or deletion operation of the release package task, the deployment of the release package in the target environment is completed.

8. The method for implementing data sandbox publishing based on automated processing as claimed in claim 7, characterized in that: After the suite service completes the creation, update or deletion operation of the release package task and the step of deploying the release package in the target environment, the method further includes: The publishing result of the publishing package is obtained, and based on the publishing result, the publishing status of the publishing package task is updated.

9. A data sandbox publishing device based on automated processing, characterized in that: The device for implementing data sandbox publishing based on automated processing comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for implementing data sandbox publishing based on automated processing as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data sandbox publishing method based on automated processing as described in any one of claims 1 to 8 are implemented.

Images